Data Cleaning Techniques and Pre-processing

• Critical data cleaning techniques and pre-processing steps are required to prepare data for analysis.
• Skills to ensure your datasets are clean, consistent, and analysis-ready are taught.

Data Cleaning Techniques

• Techniques include handling missing values and outliers.
• Missing values can arise due to non-responses in surveys, data entry errors, or system failures.
• Ignoring missing data can lead to biased analyses and misleading results.

Handling Missing Values

• Identify missing data visually or using statistical tools to detect null or NaN values.
• If the missing values are few, the affected rows or columns can be removed but do so cautiously.
• Missing data can be filled with an estimated value.
• For numerical data, use the mean or median to replace missing values.
• For categorical data, use the most frequent category (mode).
• Missing data can be predicted using other available features using regression or k-nearest neighbors (KNN).
• In time-series data, missing values can be filled using the preceding or subsequent valid data points.

Handling Outliers

• Outliers are data points significantly different from others.
• Outliers can be detected using visualization techniques like box plots or scatter plots.
• Statistical methods like Z-scores (values > 3 or < -3) or the IQR method can also be used.
• Outliers caused by data entry errors, or that are irrelevant to the analysis can be removed from the dataset.
• Transforming the data using a logarithmic transformation can help reduce the impact of extreme values.
• Capping (Winsorization) involves setting extreme values to a certain threshold (e.g the 95th percentile) to minimize the influence of outliers.
• Outliers can be imputed with more reasonable values like the median or mean, especially when they represent valid but extreme values.

Pre-processing Steps

• Raw data often contains inconsistencies, noise, or irrelevant features that can skew analysis results.
• These steps ensure that data is appropriately preprocessed, enabling accurate and reliable analysis or machine learning model development.

Data Cleaning

• Includes addressing missing values, duplicates, and outliers.
• Handling missing data can be done by imputation (filling with mean, median, mode, or prediction) or removal.
• Duplicates and outliers are removed or adjusted to avoid skewing results.

Normalization

• Scales features to a fixed range (usually [0, 1]) to ensure that all variables contribute equally.
• Is especially important for algorithms sensitive to feature magnitude, like KNN or neural networks.

Standardization

• Transforms features to have a mean of 0 and a standard deviation of 1.
• Useful when features have different scales, helping models like SVM or logistic regression converge faster.

Encoding Categorical Variables

• Converts categorical data into numerical values.
• Label encoding assigns numbers to each category.
• One-hot encoding creates binary columns for each category, making the data usable for machine learning algorithms.

Feature Engineering

• Creates new features or transforms existing ones to better represent the underlying patterns in the data.
• Binning continuous variables into categories or combining features (like height and weight to create BMI) are examples.

Handling Time-Series Data

• Converting dates into proper datetime format.
• Extracting temporal features (like day, month, or weekday).
• Helps capture patterns based on time.

Handling Imbalanced Data

• Resampling techniques or adjusting class weights can be used to prevent the model from being biased toward the majority class.
• Used when classes in a dataset are imbalanced.

Data Splitting

• Data should be split into training and testing sets.
• Ensures that the model is evaluated on unseen data.
• This helps prevent overfitting and ensures the model generalizes well to new, real-world data.