Regression Analysis & ML Algorithms with Python

Learning Outcomes

• Explanation of key regression analysis concepts.
• Instruction on evaluating regression models with different performance metrics.
• Application of various machine learning regression algorithms.
• Performance of regression tasks with Python and machine learning techniques.

Contents

• A definition of regression and its purpose.
• Explanation of model evaluation metrics.
• Covers linear regression, decision trees, and random forests.
• Teaches practical application of regression algorithms in business cases.

Regression Overview

• Regression is used to predict a value, such as the percentage of students graduating.
• It seeks to ascertain factors influencing successful graduations, such as acceptance rate, academic performance, cost of education, and cost of living.
• Regression problems use past observations, each with multiple features (variables).
• Regression determines the relationship between features to predict future values.
• The predicted feature is the dependent variable, response, or target, denoted as 'y'.
• The features used for prediction are independent variables, explanatory variables, or predictors, denoted as x1...n.

Evaluation Metrics

• et = Yt - y^t
• MAE (Mean Absolute Error) = 1/n Σ |et|
• MSE (Mean Square Error) = 1/n Σ et^2
• RMSE (Root Mean Square Error) = √(1/n Σ et^2)
• R^2 = 1 - Σet^2 / Σyt – avg(yt)
• The Evaluation Metrics steps are Load, EDA, Clean and Transform, Train, Test, and Launch.

Baseline Model

• For all test instances, the Baseline model predicts the median value seen in the training data.
• The median price is the predicted value of all observations.
• The accuracy of the baseline is measured in terms of RMSE on both the training and test sets.

Linear Regression

• In Linear Regression, a linear relationship between the dependent and independent variables is assumed.
• y = a + βx + e where 'e' denotes the residual

Linear Regressor in ML

• Linear Regression can be implemented in ML using the linear regressor.
• A linear regressor model is trained and used to fit and predict on the training set.
• The linear regressor does not appear to be an effective model for this dataset.

Decision Tree

• Decision Trees (DT) are suitable for regression and classification tasks.
• They are hierarchical models.
• The objective of a DT is to predict a target variable with simple decisions.
• DTs classify via a tree structure, beginning at the root and ending at a leaf/terminal node.

Elements of a Decision Tree

• Nodes denote conditions used to assess a particular attribute.
• Branches represent the outcomes of a condition.
• Leaves hold the final result or class label.

Advantages of Decision Trees

• DTs are easy to understand and interpret.
• They are versatile algorithms usable for both classification and regression.
• They are suitable for non-linear problems.
• DTs handle numerical and categorical variables.

Disadvantages of Decision Trees

• DTs are prone to overfitting.
• They have instability, where minor data changes can lead to different tree structures.
• Large trees are difficult to interpret.
• They have high computational cost for large trees.

Decision Tree Regressor

• Decision Tree Regressor are imported from sklearn.tree
• Decision Tree Regressor is trained using Xtrain and ytrain
• The Decision Tree predictions are more accurate than linear regression.

Random Forest

• Random Forest (RF) is an ensemble, tree-based method.
• Ensemble methods combine simple models to create a powerful model.
• A RF is composed of combining many decision trees.
• Random Forests draw multiple random samples, with replacement, from the data.
• The sampling approach of RF is called the bootstrap.
• RF creates numerous trees for a single problem, averaging the values (called aggregation).
• Bootstrapping addresses the issue of needing numerous training sets by sampling the training set with replacement.
• Bootstrapping + aggregation = Bagging

Random Forest Regressor

• Random Forest Regressor imported from sklearn.ensemble
• Train the Random Forest model using Xtrain and ytrain
• Performs better than Linear Regression model, but worse than Decision Tree model.

K-Fold Cross Validation

• Cross-validation is critical for assessing a model's generalization ability.
• Cross-validation aims to mitigate overfitting.
• K-fold cross validation splits the training data into k parts.

K-Fold Cross Validation Process

• The model is trained on k-1 parts, and performance is assessed with the remaining part.
• Each of the K parts will be used for validation across the trials and scores
• It is important to determine which of the models is the best

Cost of Error

• Under-estimate, or
• Over-estimate
• Different errors incur different costs for different stakeholders.