UEM Mathematics-1 (BSCM103) - Curve Fitting and Method of Least Squares PDF

MATHEMATICS-1(BSCM103) Dr. SAYANTAN MANDAL (ASSOCIATE PROFESSOR) SYLLABUS:  Few Rules to remember: Always ask me if you do not understand a single thing that has been taught (Even if it is very silly or common). Do not be afraid or shy to ask questions. If you are,...

MATHEMATICS-1(BSCM103) Dr. SAYANTAN MANDAL (ASSOCIATE PROFESSOR) SYLLABUS:  Few Rules to remember: Always ask me if you do not understand a single thing that has been taught (Even if it is very silly or common). Do not be afraid or shy to ask questions. If you are, you may not learn the subject properly. Never miss your homework. Try to do it on the same day. Never miss an assignment. This might contribute to your final grade.  How to contact me: Email Id: [email protected] WhatsApp No: +91 9493595817 Any Question In Your Mind ?? Let us begin…. Module-2: Curve Fitting and Method of Least Squares Curve Fitting When observations in respect of two variables are available, very often a relation is found to exist between them. For example, height and weight of persons are interdependent, expenditure depends on income, yield of a crop depends on the amount of rainfall, production depends on price, etc. Frequently, it is found desirable to express this relationship between variables by means of some mathematical equation, representing a certain geometrical curve. The process of finding such a curve or its equation on the basis of a given set of observations is called curve fitting. Curve Fitting The process of finding such a curve or its equation on the basis of a given set of observations is called curve fitting. We list below the equations of some common types of curves: 𝑦 = 𝑎 + 𝑏𝑥 (Straight Line) 𝑦 = 𝑎 + 𝑏𝑥 + 𝑐𝑥 2 (Parabola) 𝑦 = 𝑎 + 𝑏𝑥 + 𝑐𝑥 2 + 𝑑𝑥 3 (Cubic Polynomial) 𝑦 = 𝑎𝑏 𝑥 (Exponential Curve) 𝑦 = 𝑎 + 𝑏𝑐 𝑥 (Modified Exponential Curve) 𝑦 = 𝑎𝑥 𝑏 (Geometric Curve) Curve Fitting The variables 𝑥 and 𝑦 are often referred to as independent variable and dependent variable respectively. All letters 𝑎, 𝑏, 𝑐, 𝑑 except 𝑥 and 𝑦, appearing in the above equations represent constants. The choice of the appropriate equation to be fitted to a given set of observations is often facilitated by the following rules: i. Plot the corresponding observations of 𝑥 and 𝑦 as points on a graph paper. If the pattern of points shows approximately a linear path, use the straight line. (or, if the successive differences in 𝑦 corresponding to equidistant values of 𝑥 are approximately equal, use the straight line). ii. If ln 𝑦 when plotted against 𝑥 shows a linear path, use the exponential curve. iii. If ln 𝑦 when plotted against ln 𝑥 show a linear path, use the geometric curve. Straight line Straight line is the geometrical representation of an equation of the form 𝐴𝑥 + 𝐵𝑦 + 𝐶 = 0 Where 𝐴, 𝐵, 𝐶 are constants. When 𝐵 is not zero, i.e. the equation contains a term in the equation of the 𝐴 𝐶 straight line can be solved for 𝑦, giving 𝑦 = − 𝑥 + (− ) which is of the form 𝐵 𝐵 𝑦 = 𝑎 + 𝑏𝑥 where a and b are new constants. This is the form we shall generally use to represent a straight line. Sometimes, the form 𝑦 = 𝑚𝑥 + 𝑐 is also used. Note that the equations 𝑦 = 𝑎 + 𝑏𝑥 and 𝑦 = 𝑚𝑥 + 𝑐 are of the same form. The left hand side contains only 𝑦 and the right hand side contains a constant and another term involving 𝑥. Straight line Straight line Note that the equation of a straight line contains terms involving only single power of one or both the variables and a constant term. 1 There must not be any term involving (say) 𝑥2, 𝑥𝑦, 𝑥, , etc. 𝑥 Here, equations (iv) to (vii) are of the form 𝐴𝑥 + 𝐵𝑦 + 𝐶 = 0,and (i), (ii), (iii), (viii) belong to the type 𝑦 = 𝑎 + 𝑏𝑥. Equations (ix) and (x) may also be brought to any of these two forms. In the equation 𝑦 = 𝑎 + 𝑏𝑥, the coefficient of 𝑥 on the right, viz. b, is called the ‘slope’ of the straight line. The slope may be positive, negative, or zero. Problem: Find the slopes of the previously mentioned straight lines. Straight line Straight lines with positive, negative, or zero slopes. When the slope is positive, 𝑦 increases as 𝑥 increases; When the slope is negative, 𝑦 decreases as 𝑥 increases; When the slope is zero, 𝑦 remains a constant whatever be the value of 𝑥. Slope represents the amount of change (increase or decrease) in the value of 𝑦 for a unit increase in the value of 𝑥. Conversely, if it is found that the change in 𝑦 per unit increase in 𝑥 is always the same, then the relation between 𝑥 and 𝑦 can be given by an equation of the form 𝑦 = 𝑎 + 𝑏𝑥. Straight line In the equation 𝑦 = 3 + 2𝑥, for 𝑥 = 0, 1, 2, 3, the corresponding values are 𝑦 = 3, 5, 7, 9. It is seen that the values of 𝑦 increase, as 𝑥 increases; Because the slope is positive. Also the increase in 𝑦 for successive values of 𝑥 is always 2, which is the slope of the straight line. In the equation 𝑦 = 6 − 2𝑥, for 𝑥 = 0, 1, 2, 3, the values of 𝑦 are 6, 4, 2, 0. That is, the values of 𝑦 decrease, as 𝑥 increases, the successive changes being −2, which is the slope. Straight line Geometrically, the slope depends on the inclination of the straight line with the 𝑥 −axis. Two parallel straight lines have the same slope. When the slope is zero, the straight line is parallel to the 𝑥 −axis. As the slope increases, the inclination also increases. When the slope is positive, the straight line is inclined towards the right; when the slope is negative, the line is inclined towards the left. Straight line Some useful information about straight lines are: The constant term 𝑎 on the right of the equation 𝑦 = 𝑎 + 𝑏𝑥 is called the 𝑦 −intercept, i.e. the value of 𝑦 at the point where the line crosses the vertical axis (𝑥 = 0). The quantity 𝑏 is called the slope. If a straight line 𝑦 = 𝑎 + 𝑏𝑥 passes through the point (𝑥0 , 𝑦0 ), the co-ordinates must satisfy the equation, i.e. 𝑦0 = 𝑎 + 𝑏𝑥0. If a straight line passes through the points (𝑥1 , 𝑦1 ) and (𝑥2 , 𝑦2 ), 𝑦1 − 𝑦2 its slope is 𝑏 =. 𝑥1 −𝑥2 If a straight line passes through the point (𝑥0 , 𝑦0 ) and has slope b its equation is 𝑦 − 𝑦0 = 𝑏(𝑥 − 𝑥0 ) Conversely, if the equation of a straight line can be written in this form we are certain that the line passes through the point (𝑥0 𝑦0 ) and has slope 𝑏. The co-ordinates of the point of intersection of two straight lines is obtained by solving the two equations. Problem: Parabola Parabola is the geometrical representation of an equation of the form 𝑦 = 𝑎 + 𝑏𝑥 + 𝑐𝑥 2 where 𝑎, 𝑏, 𝑐 are constants the term in 𝑥 2 must be present, i.e. 𝑐 ≠ 0). The parabola is a special type of curve. In fact, the curved path followed by a flying projectile (e.g. a cricket ball thrown from a distance), is a parabola. The following equations represent parabolas: i 𝑦 = 3 − 2𝑥 + 7𝑥 2 , ii 𝑦 = 0.5𝑥 2 , iii 𝑦 = 6 − 4𝑥 2 , iv 𝑦 = 𝑎 + 𝑏𝑥 + 𝑐𝑥(𝑥 − 1). Free-hand method of curve fitting When the given data are plotted as points on a graph paper, it is often possible to draw a smooth curve through the cluster of points, which appears best to represent their pattern. The smooth curve so drawn is called a free-hand curve. It may be noted that the free-hand curve depends entirely on individual judgement, and may either be a straight line or a curved line. If the pattern of points is linear, the equation of a straight line of the form 𝑦 = 𝑎 + 𝑏𝑥 is obtained by choosing two points on the line. Let 1, 5 and (4, 11) be two such points. Substituting for 𝑥 and 𝑦 in the equation 𝑦 = 𝑎 + 𝑏𝑥, we find the relation 5 = 𝑎 + 𝑏 and 11 = 𝑎 + 4𝑏, solving which we get 𝑎 = 3 and 𝑏 = 2. Hence 𝑦 = 3 + 2𝑥 is the equation of the fitted free-hand curve, which may be used to estimate the value of y for any given value of x. Free-hand method of curve fitting Similarly, for other types of curves, it is necessary to choose as many points on the smooth curve as there are constants in the equation. The free-hand method has the disadvantage that different individuals will get different curves and equations. Method of Least Squares Method of Least Squares is a device for finding the equation of a specified type of curve, which best fits a given set of observations. The method depends upon the Principle of Least Squares, which suggests that for the "best-fitting" curve, the sum of the squares of differences between the observed and the corresponding estimated values should be the minimum possible. Suppose, we are given 𝑛 pairs of observations (𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ), … (𝑥𝑛 , 𝑦𝑛 ), and it is required to fit a straight line to these data. The general equation of a straight line 𝑦 = 𝑎 + 𝑏𝑥 is taken, where 𝑎 and 𝑏 are constants. Any values for 𝑎 and 𝑏 would give a straight line, and once these values are obtained, an estimate of 𝑦 can be obtained by substituting the values of 𝑥. That is to say, the estimated values of 𝑦 when 𝑥 = 𝑥1 , 𝑥2 , … 𝑥𝑛 would be more 𝑎 + 𝑏𝑥1 , 𝑎 + 𝑏𝑥2 , … , 𝑎 + 𝑏𝑥𝑛 , respectively. Method of Least Squares In order that the equation 𝑦 = 𝑎 + 𝑏𝑥 gives a good representation of the relationship between 𝑥 and 𝑦, it is desirable that the estimated values 𝑎 + 𝑏𝑥1 , 𝑎 + 𝑏𝑥2 , … , 𝑎 + 𝑏𝑥𝑛 , are, on the whole, close enough to the corresponding observed values 𝑦1 , 𝑦2 , … , 𝑦𝑛. For the best fitting straight line therefore, our problem is only to choose such values of 𝑎 and 𝑏 for the equation 𝑦 = 𝑎 + 𝑏𝑥 which will provide estimates of 𝑦 as close as possible to the observed values. This can be done in different ways. However, according to the Principle of Least Squares, the “best-fitting” equation is interpreted as that which minimizes the 𝑛 sum of the squares of differences ෍ 𝑦𝑖 − 𝑎 − 𝑏𝑥𝑖 2, 𝑖=1 i. e. 𝑦1 − 𝑎 − 𝑏𝑥1 2 + 𝑦2 − 𝑎 − 𝑏𝑥2 2 + ⋯ + 𝑦𝑛 − 𝑎 − 𝑏𝑥𝑛 2 Method of Least Squares Method of Least Squares Method of Least Squares: Geometrical Interpretations In the geometric sense, the problem of finding the best-fitting straight line is as follows: If the pairs of observations (𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ), … , (𝑥𝑛 , 𝑦𝑛 ) are plotted as points on a graph paper, and all possible straight lines are drawn on it, that straight line will be considered to be the “best-fitting” for which the sum of the squares of vertical distances PM between the plotted points P and the line AB is the least. Method of Least Squares: Fortunately, we do not have to find the values of 𝑎 and 𝑏 (or geometrically select the straight line) by trial. The problem is tackled by mathematical methods. This leads to a set of equations, called normal equations, solving which we get the desired values of 𝑎 and 𝑏. Similarly, the method of least squares can be used to fit other types of curves, e. g. parabola, exponential curve, geometric curve, etc. Method of Least Squares is applied to find regression lines and also in the determination of trend in time series. Method of Least Squares: How the Normal Equations are Derived: Fitting straight line 𝒚 = 𝒂 + 𝒃𝒙 : Applying the Method of Least Squares,𝑛the constants a and b are so chosen as to minimize ෍ 𝑦𝑖 − 𝑎 − 𝑏𝑥𝑖 2 𝑖=1 Taking partial derivatives with respect to a and b, and equating them to zero, we get 𝜕 2 𝜕 2 ෍ 𝑦 − 𝑎 − 𝑏𝑥 = 0, ෍ 𝑦 − 𝑎 − 𝑏𝑥 = 0 𝜕𝑎 𝜕𝑏 Normal Equations: ෍ 𝑦 − 𝑎 − 𝑏𝑥 (−2) = 0, ෍ 𝑦 − 𝑎 − 𝑏𝑥 (−2𝑥) = 0 ෍ 𝒚 − 𝒂 − 𝒃𝒙 = 𝟎, ෍ 𝒙 𝒚 − 𝒂 − 𝒃𝒙 = 𝟎 These give the normal equations: ෍ 𝒚 = 𝒂𝒏 + 𝒃 ෍ 𝒙 , ෍ 𝒙𝒚 = 𝒂 ෍ 𝒙 + 𝒃 ෍ 𝒙𝟐 Normal Equations Note: The normal equations can be easily remembered in the following way: Multiply both sides of 𝑦 = 𝑎 + 𝑏𝑥 by the coefficients of 𝑎 and 𝑏, viz. 1 and 𝑥, respectively; and then sum, using σ notation. Thus Problem: Determine the equation of a straight line which best fits the following data: 𝑥 10 12 13 16 17 20 25 𝑦 19 22 24 27 29 33 37 Problem: Determine the equation of a straight line which best fits the following data: 𝑥 10 12 13 16 17 20 25 𝑦 19 22 24 27 29 33 37 Solution: Let 𝑦 = 𝑎 + 𝑏𝑥 be the equation of the best fitting straight line by method of least squares. The constants 𝑎 and 𝑏 are obtained by solving the normal equations ෍ 𝒚 = 𝒂𝒏 + 𝒃 ෍ 𝒙 , ෍ 𝒙𝒚 = 𝒂 ෍ 𝒙 + 𝒃 ෍ 𝒙𝟐. where 𝑛 is the number of pair of observations.  Normal Equations: Hence, the equation of the best-fitting straight line is Note: The First method should be employed when the values of 𝑥 and 𝑦 are small, say 𝑥 not exceeding 10 and 𝑦 not exceeding 100. The Second method will be found convenient when the values of 𝑥 exceed 10 and those of 𝑦 are not very large. When the values of 𝑥 and 𝑦 are both large, the Third method should be employed. In case, the values of the independent variable 𝑥 are found to have a common difference, we will see some special methods that are the most convenient, requiring the minimum possible calculations. Problem: Fit a straight line by the method of least squares to the following data: Age 21 42 38 64 53 61 47 Absence 4 14 10 38 19 34 17 (No. of days) Estimate the probable number of days absent when age is 40 years. Solution: Let 𝑥 denote age (in years) and 𝑦 denote the number of days absent, and 𝑦 = 𝑎 + 𝑏𝑋 be the equation of the fitted straight line, where 𝑋 = 𝑥 − 47 Note: The origin of 𝑋 should preferably be taken near the mean of the given values of 𝑥. Using the method of the squares, the normal equations for determining the values of a and b are The constants 𝑎 and 𝑏 are obtained by solving the normal equations ෍ 𝒚 = 𝒂𝒏 + 𝒃 ෍ 𝑿 , ෍ 𝑿𝒚 = 𝒂 ෍ 𝑿 + 𝒃 ෍ 𝑿𝟐. where 𝑛 is the number of pair of observations. Simplified Calculations The labor in computation can often be minimized by a transformation of one or both the variables, either by change of origin only or by changes of origin and scale both. Situation-1: If we change the origin of 𝑥 only, i.e. write 𝑋 = 𝑥 − 𝑐, where 𝑐 is an arbitrary constant, then 𝑥 is replaced by 𝑋 in the normal equations, giving the new normal equations as: ෍ 𝑦 = 𝑎𝑛 + 𝑏 ෍ 𝑋 ෍ 𝑋𝑦 = 𝑎 ෍ 𝑋 + 𝑏 ෍ 𝑋 2 Simplified Calculations Situation-2: If we change the origins of both 𝑥 and 𝑦. i.e. write 𝑋 = 𝑥 − 𝑐 and 𝑌 = 𝑦 − 𝑐 ′ , where 𝑐 and 𝑐′ are arbitrary constants, then 𝑥 and 𝑦 are replaced by 𝑋 and 𝑌 in the normal equations, giving the new normal equations as: ෍ 𝑌 = 𝑎𝑛 + 𝑏 ෍ 𝑋 ෍ 𝑋𝑌 = 𝑎 ෍ 𝑋 + 𝑏 ෍ 𝑋 2 Simplified Calculations Situation-3: In case the successive values of the independent variable 𝑥 are found to have a common difference, two special transformations are available for the cases when (i) 𝑛 is odd, and when (ii) 𝑛 is even. Simplified Calculations Situation-3: (i) when 𝑛 is odd, write 𝑥 − Central value of 𝑥 𝑢= Common difference The values of 𝑢 will be 0 at the middle; 1, 2, 3, 4, … for the successive values, and −1, −2, −3, −4, … for the preceding values Simplified Calculations Situation-3: (i) when 𝑛 is even, write 𝑥 − mean of two central values of 𝑥 𝑢= 1 (common difference) 2 The values of u will be successively 1, 3, 5, 7, … for the lower half, and −1, −3, −5, −7, … for the upper half, starting form the middle. Simplified Calculations Situation-3: If we write the equation of straight line in the form 𝑦 = 𝑎 + 𝑏𝑢, the normal equations will be ෍ 𝑦 = 𝑎𝑛 + 𝑏 ෍ 𝑢 ෍ 𝑢𝑦 = 𝑎 ෍ 𝑢 + 𝑏 ෍ 𝑢2 But since by the above transformation σ 𝑢 = 0, these equations take very simple forms ෍ 𝑦 = 𝑎𝑛 ෍ 𝑢𝑦 = 𝑏 ෍ 𝑢2 from which the values of 𝑎 and 𝑏 can easily determined. However, in all cases, finally we have to rewrite the transformed variables 𝑋, 𝑌, 𝑢 in terms of the original variables 𝑥 and 𝑦, and then simplify the result, to get the equation of the fitted straight line.  Problem: Solution: Note: Here, the successive values of 𝑥 have a common difference and 𝑛 is odd So, the transformation in situation-3 will be most suitable. Let us write the equation of the straight line in the form 𝑥−25 𝑦 = 𝑎 + 𝑏𝑢 … … … (𝑖) where 𝑢 = 5 The normal equation are ෍ 𝒚 = 𝒂𝒏 + 𝒃 ෍ 𝒖 , ෍ 𝒖𝒚 = 𝒂 ෍ 𝒖 + 𝒃 ෍ 𝒖𝟐. where 𝑛 is the number of pair of observations. Solution:  Problem: Solution: Note: Here, the successive values of 𝑥 have a common difference and 𝑛 is even So, the transformation in situation-3 will be most suitable. Let us write the equation of the straight line in the form 𝑥−27.5 𝑦 = 𝑎 + 𝑏𝑢 … … … (𝑖) where 𝑢 = 2.5 The normal equation are ෍ 𝒚 = 𝒂𝒏 + 𝒃 ෍ 𝒖 , ෍ 𝒖𝒚 = 𝒂 ෍ 𝒖 + 𝒃 ෍ 𝒖𝟐. where 𝑛 is the number of pair of observations. Problem Method of Least Squares: How the Normal Equations are Derived: Fitting parabola 𝒚 = 𝒂 + 𝒃𝒙 + 𝒄𝒙𝟐 : Here, the constants 𝑎, 𝑏, 𝑐 are so chosen 𝑛as to minimize 2 ෍ 𝑦𝑖 − 𝑎 − 𝑏𝑥𝑖 − 𝑐𝑥𝑖2 𝑖=1 Taking partial derivatives with respect to a, b, c and equating them to zero, we get 𝜕 ෍ 𝑦 − 𝑎 − 𝑏𝑥 − 𝑐𝑥 2 2 = 0, 𝜕𝑎 𝜕 ෍ 𝑦 − 𝑎 − 𝑏𝑥 − 𝑐𝑥 2 2 = 0 𝜕𝑏 𝜕 ෍ 𝑦 − 𝑎 − 𝑏𝑥 − 𝑐𝑥 2 2 = 0 𝜕𝑐 Normal Equations: These lead to the normal equations: ෍ 𝑦 = 𝑎𝑛 + 𝑏 ෍ 𝑥 + 𝑐 ෍ 𝑥 2 ෍ 𝑥𝑦 = 𝑎 ෍ 𝑥 + 𝑏 ෍ 𝑥 2 + 𝑐 ෍ 𝑥 3 ෍ 𝑥2𝑦 = a ෍ 𝑥2 + 𝑏 ෍ 𝑥3 + 𝑐 ෍ 𝑥4 Normal Equations Note: The normal equations can be easily remembered in the following way: Multiply both sides of 𝑦 = 𝑎 + 𝑏𝑥 + 𝑐𝑥 2 by the coefficients of 𝑎, 𝑏, and 𝑐, viz. 1, 𝑥, and 𝑥 2 respectively; and then sum, using σ notation. Thus Fitting Exponential curves In order to fit curve with equation of the form 𝑦 = 𝑎𝑏 𝑥 the procedure is to take logarithms of both sides and then form normal equations. For example, to fit the Exponential Curve 𝑦 = 𝑎𝑏 𝑥 , we take logarithms of both sides, obtaining log 𝑦 = log 𝑎 + b(log 𝑥) 𝑌 = 𝐴 + 𝑏𝑋 where 𝑌 = log 𝑦 , 𝐴 = log 𝑎 , 𝑋 = log 𝑥. The normal equations are ෍ 𝑌 = 𝐴𝑛 + 𝑏 ෍ 𝑋 , ෍ 𝑋 𝑌 = 𝐴 ෍ 𝑋 + 𝑏 ෍ 𝑋 2 These are solved for 𝐴 and 𝑏, and then 𝑎 = antilog 𝐴 is obtained.