Machine Learning: Data Preparation

• Machine Learning I 258440 is taught by Ing. Wilson Arrubla at the Universidad de Sucre.

Data Preparation Steps

• Integrate data from various sources.
• Eliminate irrelevant and redundant variables.
• Describe the data statistically.
• Clean atypical data.
• Handle missing values.
• Balance the dataset.
• Transform data according to the chosen machine learning method.

Variable Types in Data Preparation

• Numerical Variables (quantitative): Examples include weight, age, years in the company, salary, sales, and debt value.
• Categorical Variables (qualitative): Examples include sex (Male, Female), Marital Status (Married, Single), if they are an adult (S, N), Disease (Yes, No), Religion (Catholic, Other), and Education Level (Bachelor's, Professional, University).
• Character Strings: Dates.

Data Integration

• Create a record for each person/product/service/location with all integrated variables.
• Eliminate duplicate records (e.g., repeated customer IDs or invoice numbers).

Eliminating Irrelevant Variables

• Remove variables that are irrelevant to the mining process to reduce noise and improve model performance.
• Delete redundant variables to simplify the dataset and avoid multicollinearity.
• Irrelevant information includes: name, phone number, address and identification document.
• Redundant variables: age, birth year, and if they are an adult. Also, sales in 1, 3 and 6 months.

Statistical Description of Data

• Use descriptive statistics and visualizations like histograms, box plots, and scatter plots for numerical variables.
• Variables are described through descriptive statistics and visualizations with histograms, frequency boxes and dispersion.

Histogram Types

• Leptokurtic: Data has many values that converge to the mean.
• Mesokurtic: Data is close to the mean.
• Platykurtic: Data has few values converging to the mean, with a large standard deviation.
• Symmetrical Distribution: mean, median and mode are equal.
• Asymmetrical Positive Distribution: mode < median < mean
• Asymmetrical Negative Distribution: mean < median < mode

Categorical Variable Representations

• Use bar charts and pie charts to visualize categorical data.

Cleaning Atypical Data (Outliers)

• Outliers are values outside the expected range.
• Establish cleaning rules to address these outliers.
• Example age range is considered 18 to 90.
  • Minimum value: 19.
  • Maximum value: 75.
  • Mean: 35.546.
  • Standard Deviation: 11.375.
• If outliers are detected, assign a NULL value.

Handling Missing Values

• Missing values can originate from various sources, including typographical errors.

Strategies to handle null values include:

• Eliminate records or variables with too many null values, rows with 30% or Columns with 30% nulos.
• Impute missing values using the mean or mode.
• Predict missing values using data mining techniques.

Balancing Data (Classification)

• Undersampling: Reduces the quantity of the majority class to balance the dataset.
• Oversampling: Randomly duplicates the minority class or creates synthetic data.

SMOTE Technique

• Synthetic Minority Oversampling Technique (SMOTE) addresses imbalanced data by oversampling the minority class.
• It generates synthetic instances to balance the dataset.
• Requires the selection of neightborhood of between 2-5 registers.
• A random register is taken and synthetic data is created.

Data Type Transformation

• This depends on the Machine Learning Method.

Data Transformation Based on ML Method Type

• Numerical ML: Normalization is used to convert data to a standard range to prevent certain features to dominate others.
  • Normalization of variables.
    • Converts categorical variables to numbers.
    • Formula: Xnorm = (X - Xmin) / (Xmax - Xmin)
• Categorical ML: Variables are categorized through Descretization. Also, Dummy Variables are created.
  • Unify ranges of variables.
  • Encode labels.
  • Create Dummies (Predictor Variables).
    • If 2 categories, one must be deleted. -If 3 or more categories exist in the model, all should be deleted and rewritten.
• For Categorical Variables the following can be used: Arbol, Bayes and Apriori.