Data Preprocessing Techniques

Data Preprocessing Overview

• Real-world databases often contain noisy, missing, and inconsistent data due to their large size and heterogeneous sources.
• Data quality depends on factors like accuracy, completeness, consistency, timeliness, believability, and interpretability.
• Data preprocessing aims to improve data quality and mining efficiency.

Data Preprocessing Techniques

• Data cleaning removes noise and corrects inconsistencies.
• Data integration merges data from multiple sources into a coherent store.
• Data reduction reduces data size by aggregation or feature elimination.
• Data transformation scales data to a smaller range, like 0.0 to 1.0

Data Cleaning Methods

• Data cleaning addresses incomplete, noisy, and inconsistent data.
• Data scrubbing tools use domain knowledge to detect and correct errors.
• Data auditing tools analyze data to discover rules or relationships and to detect violations.
• Discrepancy detection involves metadata analysis and the use of unique, consecutive, and null rules.

Handling Missing Values

• Tuples with missing class labels can be ignored, but this is ineffective if many attributes are missing values.
• Manual filling is time-consuming with large datasets
• Using a global constant is simple, but can create the false impression of an interesting result.
• Central tendency measures (mean/median) can fill missing values, with median being preferable for skewed data.
• Most probable values can be determined using regression or decision tree induction.
• Note: missing data does not always imply an error

Data Smoothing Techniques

• Binning smooths data by consulting value neighborhoods
• Smoothing by bin means replaces each bin value with the bin's mean
• Smoothing by bin medians replaces each bin value with the bin's median.
• Smoothing by bin boundaries replaces values with the closest boundary value.
• Regression conforms data values to a function.
• Outlier analysis detects outliers using clustering.

Data Integration Challenges

• Schema integration and object matching are tricky, requiring metadata analysis to avoid errors.
• Ensure attribute functional dependencies and referential constraints match across systems.

Redundancy and Correlation Analysis

• Redundancy occurs when attributes can be derived from others or inconsistencies in naming appear.
• Correlation analysis measures how strongly one attribute implies another
• Nominal data correlations can be found using the χ2 test
• Numeric data correlations can be found using the correlation coefficient
• Covariance measures the extent to which two attributes change together.

Data Reduction Strategies

• Data reduction techniques aim to obtain a smaller data representation that yields similar analytical results.
• Dimensionality reduction reduces the number of attributes.
• Numerosity reduction lowers data volume via parametric or nonparametric methods.
• Data compression applies transformations for a reduced or "compressed" data representation.
• Lossless data reduction means the original data can be built from the compressed data without the loss of any information

Wavelet Transforms

• The discrete wavelet transform (DWT) transforms a data vector into wavelet coefficients
• Useful when data is truncated after transformation
• The strongest coefficients allow for a compact, approximate version of the data
• DWT is related to the discrete Fourier transform, involves smoothing then detailing data
• Transforms data using hierarchical pyramid algorithm

Principal Components Analysis (PCA)

• PCA searches for k orthogonal vectors to represent data, projecting onto a smaller space
• Input data is normalized
• PCA Computes k orthonormal vectors
• Provides a basis for the normalized input data
• Principal components are sorted by significance
• Smaller components get weakes and those get cut

Attribute Subset Selection

• Attribute subset selection can help reduce processing overhead
• Involves removing redundant attributes or dimensions
• Aims to find minimum attribute sets
• Goal is to retain original distribution of data classes
• Employs heuristic methods that explore reduced search spaces.

Attribute Subset Selection Methods

• Stepwise forward selection starts empty, adding best attributes on each step.
• Stepwise backward elimination starts with all attributes, removing the worst on each step.
• Decision tree induction chooses the "best" attribute to partition data into classes.

Parametric Data Reduction

• Regression Approximates Data with Function Fit
• Estimates Data
• Is most useful with numeric attributes
• Log-Linear Models
• Approximate discrete multidimensional probibalility functions

Nonparametric Data Reduction

• Parametric methods store model parameters, while nonparametric methods store reduced data representations
• Histograms use binning to approximate data distributions by partitioning data into buckets of value ranges.
• Clustering groups similar data tuples into clusters, replacing the data with cluster representations
• Sampling selects a random subset of the data, trading accuracy for speed

Data Transformation

• Data transformation strategies include smoothing, attribute construction, aggregation, normalization, and discretization
• Normalization scales attribute data to a smaller range
• Discretization replaces raw values with interval or conceptual labels.
• Is crucial for mining data

Data Transformation; Normalization

• Normalization prevents attributes with larger ranges from dominating
• Min-max normalization transforms data to a range, like [0,1]
• Z-score normalization normalizes based around the man / average and standard distribution
• Decimal scaling normalizes by moving the decimal point

Discretization Methods for Numeric Data

• Binning discretizes by equal-width or equal-frequency intervals.
• Histogram analysis partitions values into disjoint ranges called buckets.
• Cluster analysis groups numeric attribute values into clusters.
• Decision tree analysis selects split-points that minimize entropy for classification.
• Correlation analysis uses measures of correlation to choose discretization intervals.

Concept Hierarchies for Nominal Data

• May often involve lots of attributes, that experts may need to pick and choose
• Concept Hierarchies may need to be explicitly expressed at the schema level
• Concept Hierarchies may be manually specified

Hierarchical Order of Attributes

• High-level concepts have fewer distinct values than lower-level concepts
• A complete hierarchy will have tightly semantically coupled attributes