Statistical Modeling and Model Selection

A statistical technique used to find the most important features in a dataset by identifying linear combinations of variables with the largest variance.

A technique that uses new predictors, which are linear combinations of the original predictors, to improve the accuracy of a regression model.

A mathematical method used to find the best linear combination of variables by minimizing the variance of the residuals.

A process of selecting the best values for the tuning parameters of a model by minimizing the loss function.

A statistical technique used to evaluate the performance of a machine learning model on unseen data by splitting the data into training and testing sets.

The discrepancy between the predicted and actual values in a regression model.

A statistical technique used to analyze a dataset and identify the most significant variables that contribute to the outcome.

A fundamental principle in machine learning that aims to balance the complexity of a model with its ability to generalize to unseen data.

A statistical measure that estimates the test error of a model by penalizing model complexity. It balances model fit (R-squared) with the number of predictors (p). A lower Cp value indicates a better model.

A criterion used to select the best model by balancing model fit and complexity. It considers the number of parameters (k) in the model and the maximum value of the likelihood function (L). Lower AIC values indicate a better model.

A criterion for model selection that aims to balance goodness of fit (R-squared) with model complexity. It penalizes models with more parameters. A higher adjusted R-squared indicates a better model.

A criterion for model selection that penalizes models with more parameters. It aims to select a model with a balance between goodness of fit and complexity. Lower BIC values indicate a better model.

Refers to the process of using a statistical model to estimate the performance of the model on unseen data, also known as test error, by evaluating its performance on a separate set of data.

The difference between the training error (error on the data used to fit the model) and the test error (error on unseen data). It indicates the extent to which a model overfits the data.

A statistical measure that estimates the test error of a model by taking into account the size of the training dataset and the number of predictors in the model. Lower values generally indicate a better model.

A piecewise polynomial function where each segment is a polynomial of a specified degree.

Points where the different polynomial segments of a piecewise polynomial function meet.

Ensuring that the piecewise polynomial function has continuous derivatives up to a certain order at the knots. This makes the function smooth and avoids sudden jumps or sharp corners.

A type of piecewise polynomial function where each segment is a linear function. They are the simplest form of splines.

A type of piecewise polynomial function where each segment is a cubic function. They offer more flexibility and smoothness compared to linear splines.

A type of cubic spline where the function extrapolates linearly between the boundary knots. This makes the spline behave more predictably at the edges.

The process of using data points to create a smooth and continuous curve that best fits the data. The curve is often represented by a spline.

A spline that is designed to minimize the overall curvature or variation of the curve while still fitting the data points. They are typically used to smooth out noisy data.

A decision tree is a supervised learning approach that uses a tree-like model of decisions and their possible consequences to visually represent the relationships between features and the target variable.

In a decision tree, internal nodes represent features used to make decisions. This means they are the points where the decision is made based on various factors represented as features.

The process of splitting the predictor space into non-overlapping regions based on the values of features. It divides the data based on certain criteria to create distinct groups.

The goal of splitting the predictor space is to find regions with minimal association, meaning the observations within a region are as similar as possible in terms of the target variable.

A top-down, greedy approach for building decision trees. It starts at the root node and makes the best decision at each step, without considering future consequences.

The Residual Sum of Squares (RSS) is used to measure the error of a decision tree. It calculates the sum of squared differences between predicted and actual values.

Pruning a decision tree involves removing branches that do not significantly contribute to the predictive ability. It makes the tree more concise and reduces overfitting.

In a classification tree, each observation is classified based on the dominant class in the region it belongs to. This means the region with the most observations of a particular class will dictate the classification.

A machine learning technique that sequentially builds multiple weak models (usually decision trees) to improve the accuracy of a prediction model.

The process of fitting a tree to the residuals of the previous model. This means the new tree focuses on the data points that the previous models didn't predict well.

A parameter that controls the learning rate in boosting. A smaller shrinkage value means the model learns more gradually.

A method to determine the optimal number of trees (B) in boosting. It involves splitting the data into training and validation sets and evaluating the performance of the model with different numbers of trees.

A measure of the importance of variables in boosting. It calculates the total decrease in RSS (residual sum of squares) for each split in a tree.

A boosting algorithm that uses regression trees. Each tree tries to capture the signal missed by the previous trees, and the final prediction is obtained by averaging the predictions of all the trees.

A similar approach to boosting. Each tree is constructed randomly, ensuring each tree tries to capture a different aspect of the data.

Similar to boosting, but the trees are constructed with perturbations based on partial residuals. The final prediction is the average of all the trees.

Gini impurity measures the homogeneity of a node in a decision tree. If all the data points in a node belong to the same class, the Gini impurity is zero. If the data points are evenly distributed across classes, the Gini impurity is high.

Bagging, short for Bootstrap Aggregating, is an ensemble learning technique that involves creating multiple decision trees on bootstrap samples of the data. The final prediction is made by averaging the predictions of all the trees.

Bootstrap sampling is a technique used to estimate the sampling distribution of a statistic. It involves repeatedly sampling with replacement from the original dataset. Each sample is called a bootstrap sample.

In random forests, a random subset of predictors is chosen for each split in the tree. The number of predictors chosen is typically a fraction of the total number of predictors. This helps to decorrelate the trees and reduce variance.

Bootstrap aggregating can improve the accuracy of a decision tree model by reducing the variance of the predictions. This is achieved by creating multiple trees on bootstrap samples of the data and averaging the predictions of all the trees.

In random forests, the chosen subset of predictors for each split is usually a fraction of the total number of predictors. For example, if there are 10 predictors, you might choose 3 predictors randomly for each split.