4

Chapter 4: Further Development and Analysis of the Classical Linear Regression Model

• This chapter delves deeper into the classical linear regression model.
• It progresses from the simple model to the more complex multiple linear regression.

Generalising the Simple Model to Multiple Linear Regression

• A simple regression model uses only one independent variable.
• Multiple linear regression models consider more than one independent variable.
• Examples of factors influencing car sales could be price of cars, public transport, petrol prices, or global warming concerns.
• Similarly, stock returns depend on several factors.

Multiple Regression and the Constant Term

• The multiple linear regression model is represented mathematically.
• The constant term (a) is often represented by a column of ones (x₁).
• A general multiple regression equation: y₁ = β₁ + β2x2₁ + β3x3₁ + ... + βkxk₁ + u₁
• Where y is the dependent variable, x₂ to xk are independent variables, and u is the error term.

Different Ways of Expressing the Multiple Linear Regression Model

• A separate equation can be written for each value of t in the multiple regression model.
• The model can be written in matrix form: y = Xβ + u.
• Matrix Breakdown:
  • y is Tx 1
  • X is Tx k
  • β is k x 1
  • u is T × 1

Inside the Matrices of the Multiple Linear Regression Model

• The constant term is often represented as a column of ones.
• Example using k=2 regressors with one column of ones.

How Do We Calculate the Parameters (the β) in this Generalised Case?

• The residual sum of squares (RSS) is minimized with respect to the coefficients (a and βs).
• In matrix notation, the RSS is given by: û'û = Σû²
• For optimal coefficients, (XX)−¹ X' y

The OLS Estimator for the Multiple Regression Model

• The OLS (Ordinary Least Squares) estimator is used to minimize the RSS.
• OLS estimate to coefficients represented as : β =(XX)-¹X'y

Calculating the Standard Errors for the Multiple Regression Model

• The standard errors of the coefficient estimates are calculated using the formula: s² = û'û/ (T – k).

Calculating Parameter and Standard Error Estimates for Different Multiple Regression Models: An Example

• An example of applying a multiple linear regression model with 15 observations.
• The sample data are used to calculate the coefficient estimates.
• The standard errors are determined by estimating the variance using RSS and the sample size.

Calculating Parameter and Standard Error Estimates for Different Multiple Regression Models: An Example (continued)

• Demonstrates the variance-covariance matrix calculation of β.
• Calculates individual variances and standard errors for each coefficient.
• A worked regression example is shown.

Testing Multiple Hypotheses: The F-test

• The t-test is used for testing single coefficients.
• An F-test is used when testing multiple coefficients simultaneously.
• It involves estimating two types of regressions (unrestricted and restricted models).

The F-test: Restricted and Unrestricted Regressions

• The 'general regression model' is shown.
• A 'hypothesis' for the coefficients is introduced.
• How the restrictions are substituted into the general model to create a 'restricted regression model'.

Calculating the F-test Statistic

• The F-statistic is a measure of comparing the unrestricted regression to the restricted regression.
• It is represented as (RRSS-URSS)/(URSS) * (T-k/m).

The F-Distribution

• The F-statistic follows the F-distribution.
• The degrees of freedom are m and (T − k).
• The relevant critical F-value is found based on the significance level and degrees of freedom.

Determining the Number of Restrictions in an F-test

• Examples for different null hypotheses
• Calculating the number of restrictions in each case.
• Alternative hypothesis for each coefficient.

What We Cannot Test with Either an F or a t-test

• This section outlines situations involving non-linear hypotheses that can't be tested using F or t-tests.

The Relationship between the t and the F-Distributions

• Hypothesis testable using F-test can also be tested with t-test. Not vice versa.
• Explains the relationship in the context of example.

F-test Example

• Provides an example, outlining the process of calculating the F-test statistic for a particular hypothesis.
• The 'hypothesised statement:' to be tested is identified.
• A full numerical calculation of the F-test is introduced, including identification of variables and the result achieved.

Data Mining

• Data mining identifies relationships in data devoid of any theoretical justification.
• A hypothetical example demonstrates the potential for significance if no theoretical background exists.

Goodness of Fit Statistics

• R² is used to measure the goodness of fit.
• R² is the square of the correlation between the predicted y-values (ŷ) and the actual y-values.
• TSS is the total sum of squares, ESS is the explained sum of squares, and RSS is the residual sum of squares.

Defining R²

• R² = ESS/TSS, which equals one minus RSS/TSS.
• Different extreme cases of R² are described and pictured.
• Issues with using R² as a measure of goodness-of-fit are outlined.

Adjusted R²

• Adjusted R² is used as a modification to solve problems with the standard R² method.
• It accounts for the loss in degrees of freedom when additional regressors are introduced.
• The formula for adjusting the R² is illustrated.

A Regression Example: Hedonic House Pricing Models

• Describes a study on housing pricing.
• The dependent variable is rental value in Canadian dollars per month.
• Several variables are used in hedonic house pricing model.

Hedonic House Pricing Models: Variable Definitions

• Defines variables used in the hedonic house price example.
• Variables like age, number of bedrooms, and amenities affect the rental price.

Hedonic House Price Results

• Presenting results from the hedonic house price analysis.
• Coefficient values, t-ratios, and expected signs.

Tests of Non-nested Hypotheses

• Explains cases where models are not nested.
• A hybrid model is proposed to test non-nested models.

Quantile Regression - Background

• Standard regression focuses on the mean (conditional mean), which isn't suitable for all cases.
• Quantile regression models the entire conditional distribution, not just the mean.

Quantile Regression - Background 2

• Quantile regressions are performed by considering several conditional quantile functions.

Quantile Regression - Background 3

• Quantile regression is a non-parametric technique which doesn't require any distributional assumptions.
• Important in financial modelling of 'tail behaviour'.
• Popular in risk management.

Quantiles - A Definition

• Quantiles are values within an ordered series (e.g. y).
• Provides mathematical definitions and examples.

Estimation of Quantile Functions

• OLS estimates the mean.
• Quantile regressions minimize the weighted sum of absolute values.

Estimation of Quantile Functions 2

• A mathematical representation of the minimisation problem of quantile functions is shown.
• The equations outline the general approach for calculating quantile functions given different quantile values in the distribution.

Quantile Regression - How not to do it

• Partitioning data and running separate regressions may lead to bias.
• Quantile regression uses the entire data set.

Quantile Regression Example

• A study examines style attribution.
• Shows how performance and exposure to various styles can be analysed using quantile regressions.

Quantile Regression Example - Discussion of Results

• Discusses the outcome of a quantile regression.
• A simple example uses the mean result and median result.

Quantile Regression Example - Table of Results

• This section presents table results of OLS and quantile regressions that were used in the study.

Quantile Regression Example - Discussion of Results 2

• Analysis of the relationship between the mean and quantile results.
• The interpretations when the results are in different quantiles are introduced (e.g., different loadings in large growth quantiles).

Factor Models and Principal Components Analysis

• Factor models reduce dimensionality in datasets with many correlated variables.
• Two types exist: economic factor models and mathematical factor models.
• Principal Components Analysis (PCA) is presented as a common mathematical approach for dimensionality reduction.

How PCA Works

• PCA transforms correlated variables into independent components.
• Explains the approach to PCA which is to transform the initial correlated variables into orthogonal principal components.
• This method explains the mathematical process.

PCA - More Details

• The importance and usefulness of components is described.
• Explains the mathematical process and resulting implications.

Principal Components as Eigenvalues

• PCA coefficients are identified as eigenvalues of X'X.
• Describes how eigenvalues, associated to principal components, summarise proportion of variation from original data.
• This demonstrates the mathematical principle and interpretations.

Principal Components as Eigenvalues

• The regression equation derived from PCA, focusing on the first few principal components, is presented.
• The principal component coefficients are shown to be linear combinations of the original OLS estimates
• The resulting interpretations are outlined.

PCA Example: An Application to Interest Rates

• Describes a study on interest rates.
• A variety of interest rates during a period were investigated.

PCA Example: The Principal Components

• The eigenvalues identify the most important elements (principal components).
• Explains how these important principal components are derived from the dataset.
• The percentage variability for each component is calculated.

PCA Example: The Factor Loadings

• Describes the factor loadings presented in tables.
• Explains how this relates to the correlation between the interest rates and main components

PCA Example: The Factor Loadings 2

• Explains how the characteristics of Dutch interest rates affect their factor loadings.
• Explains the interpretations of the observed loadings.

PCA Example: The Factor Loadings Presented

• This section presents a table of factor loadings (aj1 and aj2), for several different financial instruments/debt instruments.

Limitations of PCA

• In PCA, if you change the units of measurement, the principal component results will change too.
• Usually all variables are standardised with zero mean and unit variance prior to analysis.
• The principal components themselves don't usually have direct interpretations.

Test your knowledge of multiple linear regression with this quiz. It covers key terms, components, and principles of the multiple linear regression model. Perfect for students and enthusiasts looking to deepen their understanding of regression analysis.