Chapter 1 Introduction To Regression Analysis PDF

Chapter 1 Introduction Linear models play a central part in modern statistical methods. On the one hand, these models can approximate a large amount of metric data structu...

Chapter 1 Introduction Linear models play a central part in modern statistical methods. On the one hand, these models can approximate a large amount of metric data structures in their entire range of definition or at least piecewise. Linear Models and Regression Analysis Suppose the outcome of any process is denoted by a random variable y , called as dependent (or study) variable, depends on k independent (or explanatory) variables denoted by X 1, X 2 ,..., X k. Suppose the behaviour of y can be explained by a relationship given by y  f ( X 1, X 2 ,..., X k , 1 ,  2 ,...,  k )   where f is some well-defined function and 1 ,  2 ,...,  k are the parameters which characterize the role and contribution of X 1, X 2 ,..., X k , respectively. The term  reflects the stochastic nature of the relationship between y and X 1, X 2 ,..., X k indicates that such a relationship is not exact in nature. When   0, then the relationship is called the mathematical model otherwise the statistical model. The term “model” is broadly used to represent any phenomenon in a mathematical framework. A model or relationship is termed as linear if it is linear in parameters and nonlinear if it is not linear in parameters. In other words, if all the partial derivatives of y with respect to each of the parameters 1 ,  2 ,...,  k , are independent of the parameters, then the model is called a linear model. If any of the partial derivatives of y with respect to any of the 1 ,  2 ,...,  k is not independent of the parameters, then the model is called as nonlinear. Note that the linearity or non-linearity of the model is not described by the linearity or nonlinearity of explanatory variables in the model. For example y  1 X 12   2 X 2  3 log X 3   is a linear model because y / i , (i  1, 2,3) are independent of the parameters  i , (i  1, 2,3). On the other hand, y  12 X 1   2 X 2   3 log X   is a nonlinear model because y / 1  2 1 X 1 depends on 1 although y /  2 and y / 3 are independent of any of the 1 ,  2 or  3. Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 1 When the function f is linear in parameters, then y  f ( X 1 , X 2 ,..., X k , 1 ,  2 ,...,  k )   is called a linear model and when the function f is nonlinear in parameters, then it is called a nonlinear model. In general, the function f is chosen as f ( X 1 , X 2 ,..., X k , 1 ,  2...,  k )  1 X 1   2 X 2 ...   k X k to describe a linear model. Since X 1 , X 2 ,..., X k are pre-determined variables and y is the outcome, so both are known. Thus the knowledge of the model depends on the knowledge of the parameters 1 ,  2 ,...,  k. The linear statistical modelling essentially consists of developing approaches and tools to determine 1 ,  2 ,...,  k in the linear model y  1 X 1   2 X 2 ...   k X k   given the observations on y and X 1, X 2 ,..., X k. Different statistical estimation procedures, e.g., method of maximum likelihood, principal of least squares, method of moments etc. can be employed to estimate the parameters of the model. The method of maximum likelihood needs further knowledge of the distribution of y. In contrast, the method of moments and the principle of least squares do not need any knowledge about the distribution of y. The regression analysis is a tool to determine the values of the parameters given the data on y and X 1, X 2 ,..., X k. The literal meaning of regression is “to move in the backward direction”. Before discussing and understanding the meaning of “backward direction”, let us find which of the following statement is correct: S1 : model generates data or S 2 : data generates model. Obviously, S1 is correct. It can be broadly thought that the model exists in nature but is unknown to the experimenter. When some values to the explanatory variables are provided, then the values for the output or study variable are generated accordingly, depending on the form of the function f and the nature of the phenomenon. So ideally, the pre-existing model gives rise to the data. Our objective is to determine the functional form of this model. Now we move in the backward direction. We propose to first collect the data Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 2 on study and explanatory variables. Then we employ some statistical techniques and use this data to know the form of function f. Equivalently, the data from the model is recorded first and then used to determine the parameters of the model. The regression analysis is a technique which helps in determining the statistical model by using the data on study and explanatory variables. The classification of linear and nonlinear regression analysis is based on the determination of linear and nonlinear models, respectively. Consider a simple example to understand the meaning of “regression”. Suppose the yield of the crop ( y ) depends linearly on two explanatory variables, viz., the quality of fertilizer ( X 1 ) and level of irrigation ( X 2 ) as y  1 X 1   2 X 2  . There exist the true values of 1 and  2 in nature but are unknown to the experimenter. Some values on y are recorded by providing different values to X 1 and X 2. There exists some relationship between y and X 1 , X 2 which gives rise to a systematically behaved data on y , X 1 and X 2. Such a relationship is unknown to the experimenter. To determine the model, we move in the backward direction in the sense that the collected data is used to determine the parameters 1 and  2 of the model. In this sense, such an approach is termed as regression analysis. The theory and fundamentals of linear models lay the foundation for developing the tools for regression analysis that are based on valid statistical theory and concepts. Steps in regression analysis Regression analysis includes the following steps:  Statement of the problem under consideration  Choice of relevant variables  Collection of data on relevant variables  Specification of model  Choice of method for fitting the data  Fitting of model  Model validation and criticism  Using the chosen model(s) for the solution of the posed problem. These steps are examined below. Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 3 1. Statement of the problem under consideration: The first important step in conducting any regression analysis is to specify the problem and the objectives to be addressed by the regression analysis. The wrong formulation or the wrong understanding of the problem will give the erroneous statistical inferences. The choice of variables depends upon the objectives of the study and understanding of the problem. For example, the height and weight of children are related. Now there can be two issues to be addressed. (i) Determination of height for a given weight, or (ii) determination of weight for a given height. In case 1, the height is a response variable, whereas weight is a response variable is case 2. The role of explanatory variables is also interchanged in cases 1 and 2. 2. Choice of relevant variables: Once the problem is carefully formulated and objectives have been decided, the next question is to choose the relevant variables. It has to be kept in mind that the correct choice of variables will determine the statistical inferences correctly. For example, in an agricultural experiment, the yield depends on explanatory variables like quantity of fertilizer, rainfall, irrigation, temperature etc. These variables are denoted by X 1 , X 2 ,..., X k as a set of k explanatory variables. 3. Collection of data on relevant variables: Once the objective of the study is clearly stated, and the variables are chosen, the next question arises is to collect data on such relevant variables. The data is essentially the measurement on these variables. For example, it is important to know how to record the data on age. For example, the data collection is on age in total complete years or date of birth is to be recorded, which can give the exact age of a specific time. Moreover, it is also important to decide that the data has to be collected on variables as quantitative variables or qualitative variables. For example, if the ages (in years) are 15,17,19,21,23, then these are quantitative values. If the ages are defined by a variable that takes value 1 if ages are less than 18 years and 0 if the ages are more than 18 years, then the earlier recorded data is converted to 1,1,0,0,0. Note that there is a loss of information in converting the quantitative data into qualitative data. The methods and approaches for qualitative and quantitative data are also different. If the study variable is binary, then logistic regression is used. If all explanatory variables are qualitative, then analysis of variance technique is used. If some explanatory variables are qualitative and others are quantitative, then analysis of covariance technique is used. The techniques of analysis of variance and analysis of covariance are the special cases of regression analysis. Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 4 Generally, the data is collected on n subjects, then y on data, then y denotes the response or study variable and y1 , y2 ,..., yn are the n values. If there are k explanatory variables, X 1 , X 2 ,.., X k then xij denotes the i th value of j th variable. The observation can be presented in the following table: Notation for the data used in regression analysis Observation Response Explanatory variables Number y _____________________________________________________________ X1 X2  Xk 1 y1  x11 x1kx12 2 y2  x21 x22 x2 k 3 y3  x31 x3kx32       n yn xn1 xn 2  xnk ______________________________________________________________________________________ 4. Specification of the model: The experimenter or the person working in the subject usually help in determining the form of the model. Only the form of the tentative model can be ascertained, and it will depend on some unknown parameters. For example, a general form will be like y  f ( X 1 , X 2 ,..., X k ; 1 ,  2 ,...,  k )   where  is the random error reflecting mainly the difference in the observed value of y and the value of y obtained through the model. The form of f ( X 1 , X 2 ,..., X k ; 1 ,  2 ,...,  k ) can be linear as well as nonlinear depending on the form of parameters 1 ,  2 ,...,  k. A model is said to be linear if it is linear in parameters. For example, y  1 X 1   2 X 12  3 X 2   y  1   2 ln X 2   are linear models whereas y  1 X 1   22 X 2  3 X 2   y  ln 1 X 1   2 X 2   are the non-linear models. Many times, the nonlinear models can be converted into linear models through some transformations. So the class of linear models is wider than what it appears initially. Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 5 If a model contains only one explanatory variable, then it is called a simple regression model. When there are more than one independent variables, then it is called a multiple regression model. When there is only one study variable, the regression is termed as univariate regression. When there are more than one study variables, the regression is termed as multivariate regression. Note that the simple and multiple regressions are not same as univariate and multivariate regressions. The choice between simple and multiple regression is determined by the number of explanatory variables, whereas the choice between univariate and multivariate regressions is determined by the number of study variables. 5. Choice of method for fitting the data: After the model has been defined, and the data have been collected, the next task is to estimate the parameters of the model based on the collected data. This is also referred to as parameter estimation or model fitting. The most commonly used method of estimation is called the least-squares method. Under certain assumptions, the least-squares method produces estimators with desirable properties. The other estimation methods are the maximum likelihood method, ridge method, principal components method etc. 6. Fitting of the model: The estimation of unknown parameters using appropriate method provides the values of the parameter. Substituting these values in the equation gives us a usable model. This is termed as model fitting. The estimates of parameters 1 ,  2 ,...,  k in the model y  f ( X 1 , X 2 ,..., X k , 1 ,  2 ,...,  k )   are denotes as ˆ1 , ˆ2 ,..., ˆk which gives the fitted model as y  f ( X 1 , X 2 ,..., X k , ˆ1 , ˆ2 ,..., ˆk ). When the value of y is obtained for the given values of X 1 , X 2 ,..., X k , it is denoted as ŷ and called as fitted value. The fitted equation is used for prediction. In this case, ŷ is termed as the predicted value. Note that the fitted value is where the values used for explanatory variables correspond to one of the n observations in the data, whereas predicted value is the one obtained for any set of values of explanatory variables. It is not generally recommended to predict the y -values for the set of those values of explanatory variables which lie for outside the range of data. When the values of explanatory variables are the future values of explanatory variables, the predicted values are called as forecasted values. Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 6 There are different methodologies based on regression analysis. They are described in the following table: _______________________________________________________________________________________ Type of Regression Conditions Univariate Only one quantitative response variable Multivariate Two or more quantitative response variables Simple Only one explanatory variable Multiple Two or more explanatory variables Linear All parameters enter the equation linearly, possibly after transformation of the data Nonlinear The relationship between the response and some of the explanatory variables is nonlinear or some of the parameters appear nonlinearly, but no transformation is possible to make the parameters appear linearly Analysis of variance All explanatory variables are qualitative variables Analysis of Covariance Some explanatory variables are quantitative variables and others are qualitative variables Logistic The response variable is qualitative 7. Model criticism and selection The validity of the statistical method to be used for regression analysis depends on various assumptions. These assumptions become essentially the assumptions for the model and the data. The quality of statistical inferences heavily depends on whether these assumptions are satisfied or not. For making these assumptions to be valid and to be satisfied, care is needed from the beginning of the experiment. One has to be careful in choosing the required assumptions and to decide as well to determine if the assumptions are valid for the given experimental conditions or not? It is also important to determine that the situations in which the assumptions may not meet. Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 7 The validation of the assumptions must be made before drawing any statistical conclusion. Any departure from the validity of assumptions will be reflected in the statistical inferences. In fact, the regression analysis is an iterative process where the outputs are used to diagnose, validate, criticize and modify the inputs. The iterative process is illustrated in the following figure. Inputs Outputs  Theories Estimate  Estimation of parameters  Model  Confidence regions  Data  Tests of hypothesis  Statistocal methods  Graphical displays Diagnosis,  Assumptions validation and critcisim 8. Objectives of regression analysis The determination of the explicit form of the regression equation is the ultimate objective of regression analysis. It is finally a good and valid relationship between study variable and explanatory variables. Such a regression equation can be used for several purposes. For example, to determine the role of any explanatory variable in the joint relationship in any policy formulation, to forecast the values of the response variable for a given set of values of explanatory variables. The regression equation helps in understanding the interrelationships of variables among them. Regression Analysis | Chapter 1 | Introduction | Shalabh, IIT Kanpur 8