Feature Engineering Techniques

Study Notes

Techniques for enhancing data quality and improving machine learning model performance include data imputation, data normalization, one-hot encoding, feature engineering in time series and NLP, and data dimensionality reduction.

Methods for handling missing values include using the next or previous value, K-Nearest Neighbors, maximum or minimum values, missing value prediction, most frequent values, average or linear interpolation, rounded mean or moving average, or median value, and fixed values.

Min-max normalization: Scales data to a specific range (typically 0 to 1).
- Formula: y = (x - xmin) / (xmax - xmin) where 'x' is the original value, 'xmin' is the minimum value, and 'xmax' is the maximum value.
Z-score normalization: Centers the data around a mean of zero and scales it by its standard deviation.
- Formula: y = (x - mean(x)) / stddev(x) where 'x' is the original value, 'mean(x)' is the mean, and 'stddev(x)' is the standard deviation.
Normalization by decimal scaling: Scales data to have a maximum absolute value less than 1.
- Formula: y = x / 10^j, where 'j' is the smallest integer.

Converts categorical variables into numerical representations. Replaces categories with binary vectors (e.g., Red = [1, 0, 0], Green = [0, 1, 0], Blue = [0, 0, 1]).

Applied to data for various reasons and is useful for feature scaling
Can improve normality, reduce skewness and helps handle outliers. Particularly useful on skewed datasets

Outlier detection: Methods to identify unusual data points in a data set.
Remove Outliers: Eliminating outliers from a data set
Transform outliers: Methods such as log transformations, to reduce or normalize the effect of outliers.
Imputing outliers: Replacing outliers with more typical values like means, medians, modes, or nearest neighbors.

Second-order differences: Finding differences between successive data points to determine if data is stationary.
- Second order difference: y'(t) - y'(t-1)
- Formula: y = x(t) – x(t-1) & y'(t) = y(t) - y(t-1)
Logarithm: Calculating the logarithm of a value to smooth variations in data and help achieve seasonality. Formula examples include log(y(t)) & log(y'(t)).
Seasonal-trend decomposition: a method that decomposes a time series into its constituent components: trend, seasonality, and remainder. This facilitates identifying patterns/seasonality.

Bag of words: Represents text by counting the occurrences of each word.
Term Frequency-Inverse Document Frequency (TF-IDF): Weights words by their frequency in a document and inverse frequency across the entire corpus (collection of documents). Formula: TF-IDF = TF * IDF
Word2Vec: Converts words into numerical vectors, capturing semantic relationships between words.

Techniques to select the most relevant features for a machine learning task.
- Unsupervised: Drop incomplete features/ features with high multicollinearity
- Supervised: Forward selection, backward selection, recursive feature elimination, Chi-squared tests, Mutual Information tests and Pearson's R, Kendall's Tau, Spearman's Rho and F-score features

Techniques to reduce the number of variables in data while preserving important information.
- Principal Component Analysis (PCA): Creates new uncorrelated variables (principal components) from existing ones.
- Linear Discriminant Analysis (LDA): Finds directions in a dataset that best separate between classes.
- Autoencoders: Neural networks that learn to compress and reconstruct data, resulting in a reduced representation.