Untitled Quiz

Feature Selection

• A machine learning procedure to find a subset of features for a better model.
• Aims to avoid overfitting and improve generalization.
• Reduces storage requirements and training time.
• Enhances interpretability.

Relevant vs. Redundant Features

• Feature selection keeps relevant features for learning and removes redundant and irrelevant ones.
• For example, in binary classification, feature f1 might be relevant, f2 redundant given f1, and f3 irrelevant.
• Visualizations (graphs) show the distinction between relevant, redundant, and irrelevant features in binary classification tasks.

Feature Selection vs. Feature Extraction

• Feature extraction creates new features from existing ones, while feature selection chooses a subset of existing ones.
• Feature selection preserves the original features' meaning for better model interpretability.

Interpretability of Learning Algorithms

• Feature selection enhances the accuracy and interpretability of many learning algorithms.

When Feature Selection is Important

• Dealing with noisy data.
• Handling many low-frequency features.
• Using multi-type features.
• Having too many features compared to samples.
• Working with complex models.
• Dealing with inhomogeneous training and test samples in real-world scenarios.

Types of Feature Selection

• Label perspective: Supervised, unsupervised, semi-supervised.
• Selection strategy perspective: Wrapper methods, filter methods, embedded methods.

Supervised Feature Selection

• Used for classification or regression problems.
• Aims to find features that discriminate between classes or approximate target variables.
• Uses labeled data during feature selection.
• Involves a training set, feature information, selected features, supervised learning algorithm, and classifier.

Unsupervised Feature Selection

• Used for clustering problems.
• Label information is often expensive to collect (time-consuming).
• Alternative criteria for feature relevance.
• Uses an unsupervised learning algorithm.
• Involves feature information, feature selection, selected features, and unsupervised learning algorithm.

Semi-Supervised Feature Selection

• Uses both labeled and unlabeled data.
• Exploits both labeled and unlabeled data to identify relevant features.
• Involves partial label information, feature information, training set, feature information, testing set, selected features, semi-supervised learning algorithm, and classifier.

Feature Selection Techniques

• Subset selection: Forward and Backward search, greedy approach.
• Forward Search: Starts with no features and greedily adds the most relevant one until reaching the desired number.
• Backward Search: Starts with all features and greedily removes the least relevant ones until the desired number is reached.
• Inclusion/Removal criteria: Uses cross-validation.

Wrapper Methods

• Relies on the predictive performance of a given learning algorithm.
• Iteratively searches for a feature subset and evaluates its performance.
• Iteratively searches a subset of features and evaluates its performance.
• Computational expensive.
• Typically uses greedy search strategies (e.g., sequential, best-first, branch and bound).

Filter Methods

• Independent of the learning algorithm.
• Evaluates feature importance based on data characteristics (e.g., correlation, mutual information).
• More efficient than wrapper methods.
• Selected features may not be optimal for a specific learning algorithm.

Single Feature Evaluation

• Frequency-based methods.
• Dependence of feature and label (Co-occurrence).
• Mutual Information, Chi-square statistic.
• Information theory (KL divergence, Information gain).
• Gini indexing.

Embedded Methods

• A trade-off between wrapper and filter methods.
• Embeds feature selection into the learning algorithm (e.g., ID3).
• Inherits the merits of wrappers and filters (interactions with the learning algorithm).
• More efficient than wrapper methods.
• Biased toward the underlying learning algorithm.

Traditional Feature Selection

• Categorized into information-theoretic methods, statistical methods, and sparse learning methods.

Information Theoretical Methods

• Employs heuristic filter criteria to measure feature importance.
• Aims to find optimal features (relevant and non-redundant).

Preliminary Information Theoretical Measures

• Entropy of a discrete variable X.
• Conditional entropy of X given Y.
• Information gain between X and Y.
• Conditional Information Gain.

Sample Examples

• Detailed example calculations for Entropy of Y, Conditional Entropy, and Information Gain.

Mutual Information-based Feature Selection

• Information gain considers only feature relevance.
• Features should not be redundant.
• The score of a new feature fk considers both relevance and redundancy.

Minimum Redundancy Maximum Relevance

• Improves on mutual information by considering redundancy.
• The score of a new feature considers relevance and reduced redundancy.

Conditional Infomax Feature Extraction

• Feature usefulness is determined by stronger correlation within classes compared to overall correlation.
• Correlation does not imply redundancy.

g*( ) Function as Nonlinear Function

• This function can be linear or nonlinear.

Information Gain (Lewis)

• Information gain solely considers feature correlation with class labels.

Mutual Information(Battiti)

• Mutual Information considers relevance and redundancy of features.

Statistical Methods

• Based on different statistical criteria to assess features.
• Most are filter methods, evaluating features independently.
• Data discretization is often needed for numerical features.
• T-Score and Chi-Square methods are examples used for binary and multi-class classification.

Statistical Based Methods - Summary

• Includes methods like Low Variance-CFS, Kruskal Wallis.
• Pros: Computationally efficient.
• Cons: Cannot handle feature redundancy, requires data discretization.

Feature Selection Issues (Big Data)

• Data Variety, Velocity, Volume.

Feature Sparsity

• Indicates that many model parameters have small or zero values.

Sparse Learning Methods

• Framework for finding optimal features (often uses l-norm regularization).
• Examples: Lasso Regression.

Extension to Multi-Class or Multi-variate Problems

• Adapting feature selection to handle multiple target variables in classification or regression.
• Example method involves l2,1-norm.

Sparse Learning Methods - Summary

• Includes multi-label feature selection and other techniques.
• Pros: Often provides good model performance and interpretability.
• Cons: Selected features might not be suitable for other tasks, computation can be expensive due to non-smooth optimization.

Feature Engineering - Additional Techniques

• Numerical data (SVD, PCA).
• Textual data (Bag-of-Words, TF-IDF).
• Time series and GEO data.
• Image data.
• Relational data.
• Anomaly detection.