Feature Engineering Techniques

Feature Engineering Techniques

• Techniques for enhancing data quality and improving machine learning model performance include data imputation, data normalization, one-hot encoding, feature engineering in time series and NLP, and data dimensionality reduction.

Data Imputation

• Methods for handling missing values include using the next or previous value, K-Nearest Neighbors, maximum or minimum values, missing value prediction, most frequent values, average or linear interpolation, rounded mean or moving average, or median value, and fixed values.

Data Normalization

• Min-max normalization: Scales data to a specific range (typically 0 to 1).

  • Formula: y = (x - xmin) / (xmax - xmin) where 'x' is the original value, 'xmin' is the minimum value, and 'xmax' is the maximum value.

• Z-score normalization: Centers the data around a mean of zero and scales it by its standard deviation.

  • Formula: y = (x - mean(x)) / stddev(x) where 'x' is the original value, 'mean(x)' is the mean, and 'stddev(x)' is the standard deviation.

• Normalization by decimal scaling: Scales data to have a maximum absolute value less than 1.

  • Formula: y = x / 10^j, where 'j' is the smallest integer.

One-Hot Encoding

• Converts categorical variables into numerical representations. Replaces categories with binary vectors (e.g., Red = [1, 0, 0], Green = [0, 1, 0], Blue = [0, 0, 1]).

Log Transform

• Applied to data for various reasons and is useful for feature scaling
• Can improve normality, reduce skewness and helps handle outliers. Particularly useful on skewed datasets

Handling Outliers

• Outlier detection: Methods to identify unusual data points in a data set.
• Remove Outliers: Eliminating outliers from a data set
• Transform outliers: Methods such as log transformations, to reduce or normalize the effect of outliers.
• Imputing outliers: Replacing outliers with more typical values like means, medians, modes, or nearest neighbors.

Feature Engineering in Time Series Analysis

• Second-order differences: Finding differences between successive data points to determine if data is stationary.
  • Second order difference: y'(t) - y'(t-1)
  • Formula: y = x(t) – x(t-1) & y'(t) = y(t) - y(t-1)
• Logarithm: Calculating the logarithm of a value to smooth variations in data and help achieve seasonality. Formula examples include log(y(t)) & log(y'(t)).
• Seasonal-trend decomposition: a method that decomposes a time series into its constituent components: trend, seasonality, and remainder. This facilitates identifying patterns/seasonality.

Feature Engineering in Natural Language Processing (NLP)

• Bag of words: Represents text by counting the occurrences of each word.
• Term Frequency-Inverse Document Frequency (TF-IDF): Weights words by their frequency in a document and inverse frequency across the entire corpus (collection of documents). Formula: TF-IDF = TF * IDF
• Word2Vec: Converts words into numerical vectors, capturing semantic relationships between words.

Feature Selection

• Techniques to select the most relevant features for a machine learning task.
  • Unsupervised: Drop incomplete features/ features with high multicollinearity
  • Supervised: Forward selection, backward selection, recursive feature elimination, Chi-squared tests, Mutual Information tests and Pearson's R, Kendall's Tau, Spearman's Rho and F-score features

Data Dimensionality Reduction

• Techniques to reduce the number of variables in data while preserving important information.
  • Principal Component Analysis (PCA): Creates new uncorrelated variables (principal components) from existing ones.
  • Linear Discriminant Analysis (LDA): Finds directions in a dataset that best separate between classes.
  • Autoencoders: Neural networks that learn to compress and reconstruct data, resulting in a reduced representation.