Regression Analysis

Regression Analysis

• Regression analysis is used to predict the value of one variable from known values of other variables
• The variable to be predicted is the dependent or explained variable
• Variables used to predict the dependent variable are called independent or explanatory variables
• Simple regression analysis is confined to the study of only two variables, a dependent variable and one independent variable
• Simple linear regression is used when the relationship between the dependent and independent variable is linear
• Regression helps to study the functional relationship between variables for prediction or forecasting

Meaning of Regression

• The word 'regression' originally meant 'stepping back' or 'returning to average value'
• Sir Francis Galton introduced regression as a statistical concept in 1877, studying heights of fathers and sons
• Galton's studies showed that offspring of abnormally tall or short parents tend to revert toward the average height of the population

Lines of Regression - The Least Squares Approach

• This involves estimating or predicting values of a dependent variable based on known values of an independent variable
• Bivariate data consists of pairs of observations (X, Y) on two quantitative variables X and Y
• Assumes X and Y are approximately linearly related, following a straight line on a scatter diagram
• A line can be visually fitted to approximate the data and predict a value of Y for a given X
• The least squares criterion selects the line of "best fit" by minimizing the sum of the squares of vertical distances from observed points
• A line of best fit is represented by the equation Y = a + bX
• Constants a and b are determined to minimize the sum of squared vertical distances
• The determination of a and b uses differential calculus and results in two normal equations
• The normal equations are ΣY = na + bΣX and ΣXY = aΣX + bΣX²

Solving for Regression Coefficients

• Solving the normal equations simultaneously yields formulas for a and b
• The formula for b = [nΣXY - (ΣX)(ΣY)] / [nΣX² - (ΣX)²]
• The formula for a = Y − bX, where X and Y are means of X and Y, respectively
• The line of best fit is called the least squares line of regression of Y on X, where b is a regression coefficient, measuring the change in Y per unit change in X
• The line of regression of X on Y allows to estimate a value of X for a given value of Y, is given by X = c + dY
• The constant d is called the regression coefficient of X on Y, denoted by bxy
• The regression coefficient bxy measures the change in X corresponding to a unit change in Y
• The formula for d = [nΣXY - (ΣX)(ΣY)] / [nΣY² - (ΣY)²]
• The formula for c = X - dY

Equations for Regression Lines

• The equation for the line of regression of Y on X can be written as Y − Y = byx(X − X)
• The equation for the line of regression of X on Y can be written as X − X = bxy(Y − Y)
• There are always two lines of regression used depending on independent, dependent variables
• The regression line of Y on X is used to estimate Y from known values of X
• Similarly, regression line of X on Y is used to predict or estimate values of X from known values of Y
• The two regression equations are not reversible

Two Regression Lines and Perfect Correlation

• The two regression lines intersect ate the point (X, Y), that is the mean value
• The basis for deriving the two regression equations are different
• The two lines would coincide in case of perfect correlation (positive or negative) and one line is sufficient
• Two regression equations reduce to Y=Y arid X=X, if X and Y are uncorrelated and are perpendicular to each other

Regression Coefficients - Some Formulas

• The regression coefficient of Y on X, byx = Cov(X,Y) / σx²
• The regression coefficient of X on Y, bxy = Cov(X,Y) / σy²
• The covariance between X and Y is given by Cov(X,Y) = (ΣXY / n) – (ΣX ΣY / n²)
• The variance of X and Y values are respectively given by σx² = Σ(X-X)² / n and σy² = Σ(Y-Y)² / n

Further Formulas for Regression Coefficients

• byx = r (σy / σx) and bxy = r (σx / σy), where r is the coefficient of correlation
• The two regression equations can be expressed as Y−Y = r (σy / σx) (X -X) X-X=r (σy / σx) (Y -Y)
• When there is a perfect correlation (r = ±1), regression equation of Y on X becomes, (Y-Y/σy) = (X -X/σx)(or, (X -X/σx) =(Y-Y/σy)
• the two regression lines coincide in case of perfect correlation .If r = 0, i.e., if X and Y are uncorrelated, the two regression equations reduce to Y- Y and X-X, and hence they are perpendicular to each other

Properties of Regression Coefficients

• The coefficient of correlation and the two regression coefficients have the same signs, since bxy and byx depends on COV(X,Y)
• The coefficient has the same as sign that of the Cov(X,Y)
• The coefficient of correlation is the geometric mean between the regression coefficients;ie bxy and byx =r2
• If one of the regression coefficients is greater than unity, the other must be less than unity because must be less than one
• Regression coefficients are independent of change of origin but not of scale
• We know that if U=(X-A) /hand V=(Y-B) / kthen byx =bk/h and bxy=bkv/hv
• The correlation coefficient is independent of the change in origin and scale therefore rxy equals ruv

Standard Error of Estimate

• Primary use of regression is to estimate values of dependent from the independent variable.
• The reliability of such estimates depends on the closeness of the relationship between variables
• The Standard error of estimate is a measure of the extent of spread/scatter around the regression line, indicating the extent of spread or scatter of the points about the regression line
• The standard error of estimate measures the scatter or spread of the observed values relative to the values predicted by the regression line (standard deviation measuring the scatter or spread of actual values
• Syx = sqrt[ Σ(Y - Yc)² / n ], where Yc: computed/estimated Y
• Syx = sqrt[ ΣY² − aΣY − bΣXY / n ] is a convenient formula for computation
• Syx - σy√1-r2, here r = coefficient of correlation
• It indicates the extent of the possible variations (or error) that may be present; i.e. the spread of points around the regression line

Guidelines for problems

• If it is given that "n" values of the "X" and "Y" are mentioned in the data, use b = [ (nΣXY )-(ΣX)(ΣY) ]/[ nΣX² - (ΣX)²]
• However, if in the question, there are some notations given use ∑x (Summations) to solve the problems [Refer ex 2, 3, 4, 5]:

Examples for Regression Analysis

• In case since building Permits have been expressed in hundreds so x=3 if the data, then number of building permit is equal to 300
• From the examples for this chapter, you can notice that there can be different forms of questions arising in it exam

Examples of solved questions

• Example 3, 4, 5, 6 are for basic and clear understanding regression coefficient
• Example 7, 8, 9 helps one to understand "application level questions/ practical issues"
• Other later numerical and formulas are in the text

Multiple Linear Regression Analysis

• The multiple linear regression model is a more general form of the linear regression model in which the criterion variable "Y'' is specified as a function of multiple predictor variables
• The most general formula used in predicting the value is Y= B0 + B2* x1+ B2* x2+ B3* x3+ B4* x4........ Be *xn
• Where, B0 intercept, x1-to -xn different predictor variables with B2 to Be coefficients for respective predictor variablzs

Non-linear Trend/ Parabolic trend:

The equation of "Parabolic Curve" to better measure to trend is denoted with equation form: Yc = a + bx + cx² Since ΣX = 0 then a=Y/N and c = ΣXY²/ΣX²

Shifting of Orgin for parabola trend

• Yc 35+5X+ 3X2
• Here for the shift of a, b. then c remains untouched
• Yc a2 + b2x+cx²

Trend values using regression model

#NOTE: I did my best to keep to the content requested. Let me know how it is!