Data Transformation and Normalization Techniques

Data Transformation

• Data transformation involves changing data from one format to another, essential in data preprocessing.
• Methods include binning, clustering, regression, and a combination of human and computer inspection.
• Binning converts numerical data into categorical components.
• Clustering involves grouping data based on similarity.
• Regression utilizes a regression line to analyze relationships.

Normalization Techniques

• Normalization scales specific variables to fit within a small range.
  • Min-max normalization transforms values to a new scale.
  • Z-score standardization converts a numerical variable to a standard normal distribution.

Encoding and Binning

• Binning categorizes numerical variables into categorical counterparts.
  • Equal-width partitioning divides data into N intervals of equal size.
  • Equal-depth partitioning ensures each interval contains approximately the same number of samples.

Data Reduction

• Aims to obtain a condensed representation of datasets.
• Techniques include sampling and feature subset selection.

Sampling Methods

• Simple random sampling allows equal selection probability.
• Sampling without replacement does not reuse selected items.
• Sampling with replacement reuses items in the population.
• Stratified sampling divides data into various partitions for selection.

Feature Subset Selection

• Reduces dimensionality by removing redundant features.
• Techniques include:
  • Brute-force approach which tests all possible feature combinations.
  • Embedded approaches which naturally select features.
  • Filter approaches that select features based on their relevance.
  • Wrapper approaches which utilize a mining algorithm as a black box.

Data Cleaning

• Addresses anomalies in data storage before mining.
• Major tasks include filling in missing values and cleaning noisy data.
• Steps for data cleaning encompass monitoring errors, validation of data accuracy, and scrubbing duplicate data.