Data Preprocessing

• Data preprocessing prepares raw data for machine learning.
• It is the first and crucial step while creating a machine learning model.

Why Data Preprocessing Is Needed

• Real-world data often contains noises, missing values, or unusable formats.
• These issues prevent the data from being directly used for machine learning models.

Steps for Data Preprocessing

• Acquire the dataset.
• Import relevant libraries (e.g., Numpy, Pandas, Matplotlib).
• Import the dataset into the working environment or notebook.
• Identify and handle missing values.
• Encode categorical data.
• Split the dataset into training and testing sets.
• Apply feature scaling.

Acquiring the Dataset

• Involves gathering data from multiple sources into a combined format.
• Dataset formats vary based on the use case (e.g., business or medical).
• Datasets are typically stored in CSV, HTML, or XLSX file formats.

Data Types

• Useless data has no statistical relevance to the problem being solved.
• Indexes, IDs, account numbers, names, and email addresses can be considered useless data.
• Binary data has only two possible values and may include binary classification labels.
• Data domain experts and unique value counts can be used for binary data detection.
• Continuous, or 'interval', data is measured on a continuous quantified scale, such as temperature or value.
• Categorical data, the order of the data matters, but the 'distance' between values is not quantified
• Color, species and drink preference are included as nominal categorical data.
• Temporal data includes entries such as dates, time or order.

Importing Libraries

• Numpy: Fundamental package for scientific calculation in Python, used for mathematical operations and multidimensional arrays.
• Pandas: Open-source Python library for data manipulation and analysis, used for importing and managing datasets.
• Matplotlib: Python 2D plotting library for creating various types of charts.

Sample Dataset Characteristics

• Features three independent variables: Country, Age, Salary.
• Features one dependent variable: Purchased.
• Contains missing values in Age and Salary
• Includes a categorical variable: Country

Importing the Dataset

• It's recommended to save the Python file in the directory with the dataset.
• read_csv() from Pandas imports a CSV file.
• It is necessary to separate the data in the dataset into independent and dependent variables .
• Employ the iloc[] feature in the Pandas library to isolate the independent variables.

Handling Missing Values

• Identify and correctly handle missing values to avoid inaccurate inferences.
• Methods include:
  • Deleting rows: removing rows with missing values, but use with caution.
  • Imputing data: replacing missing values using mean, median, or constant values.

Imputing Data (Handling Missing Values)

• Imputation can add variance but negates data loss.
• Often yields better results as compared to omitting rows or columns.

Encoding Categorical Data

• Converts categorical information into numerical data.
• Machine learning models rely on numerical calculations.

Ordinal Data Mapping

• The column satisfaction is ordinal.
• Because order matters in this column, the mapping should reflect this order.

One-Hot Encoding (Nominal Data)

• Nominal Data is not ordered.
• Dummy Encoding is used to eliminate this issue, generating dummy variables with 0 or 1 to represent the presence of a category.

Encoding Continuous Data

• Binning.
• Mapping
• OneHotEncoding (nominal data)

Splitting the Dataset

• Datasets are divided into training and test sets.

Purpose of Splitting

• Improves model performance.
• Training sets "teach" the model.
• Test sets evaluate the model's ability to generalize.

Train Test Split

• Datasets are commonly split using a 70:30 or 80:20 ratio between the training and testing sets.
• Four elements are present in the code.
  • X_train displays features for training data.
  • X_test displays features for testing data.
  • y_train is for dependent variables during training.
  • and y_test is and independent variables for testing.

Feature Scaling (Normalization) and Binning

• Marks the end of data preprocessing in machine learning.

Feature Scaling

• Standardizes independent variables within a specific range.
• Limits the range of variables for comparison.
• Prevents algorithms from being unduly influenced by higher values.

Standardisation and Normalisation Methods

• Standardization applies transformations.
• Normalization scales values between 0 and 1.

Feature Scaling in Specific Sample Datasets

• Age and salary columns might not have the same scale so feature scaling addresses this as salary values can easily dominate age values and deliver incorrect results.

Data Binning

• A type of data smoothing.
• Groups data into bins or buckets, replacing values within a interval with single representative value for interval to convert numeric values to categorical value.
• Improves accuracy in models.

Binning Techniques

• Binning by distance: Define the edges of each bin.
• Binning by frequency: Calculates the size of each bin so that each bin contains the same number of observations by dividing the dataset into equal portions.
• *Binning by sampling (mean, median, boundary)**: Each is employed to reduce samples, by grouping similar values of contiguous values.