Lesson 5: Curve Fitting - Computational Methods in Physics I - PDF

Computational Methods in Physics I :PHY405 Lesson 5 Chapter 17 & 18 on Textbook  Plan Least Squares Regression: Linear Regression  Multiple Linear Regression General Linear Fitting with Nonlinear Functions Non-Linear Regres...

Computational Methods in Physics I :PHY405 Lesson 5 Chapter 17 & 18 on Textbook  Plan Least Squares Regression: Linear Regression  Multiple Linear Regression General Linear Fitting with Nonlinear Functions Non-Linear Regression: Linearization Method Interpolation:  Newton’s Divided Difference Method  Lagrange Interpolation Inverse Interpolation Errors in polynomial Interpolation Least Squares Regression  INTRODUCTION In many situations in physics, we need to know the global behavior of a set of data in order to understand the trend in a specific measurement or observation  Given a set of experimental data: x 1 2 3 y 5.1 5.9 6.3  The relationship between x and y may not be clear.  Find a function f(x) that best fit the data 1 2 3  Selection of the Functions Linear f ( x) = a + bx 2 Quadratic f ( x) = a + bx + cx n Polynomial f ( x) = ∑ ak x k k =0 m General f ( x ) = ∑ ak g k ( x ) k =0 g k ( x) are known. 5  Least Squares Regression Given: xi x1 x2 …. xn yi y1 y2 …. yn The form of the function is assumed to be known but the coefficients are unknown. 2 ei = ( yi − f ( xi )) 2 = ( f ( xi ) − yi ) 2 The difference is assumed to be the result of experimental error. 6  How determine the Unknowns We want to find a and b to minimize : n Φ ( a , b) = ∑ ( a + bxi − yi ) 2 i =1 How do we obtain a and b to minimize : Φ ( a, b) ? n ∂ Φ ( a , b) ∂ a = ∑ 2(a + bxi − yi ) = 0 i =1 n ∂ Φ ( a , b) ∂ b = ∑ 2(a + bxi − y i )xi = 0 i =1 7 Normal Equations Solving the Normal Equations 8 8  Linear Regression Assume : f(x) = a + bx Equations :  n   n  n a +  ∑ xi  b =  ∑ yi   i =1   i =1   n   n 2  n   ∑ xi  a +  ∑ xi  b =  ∑ xi y i         i =1   i =1   i =1  9 Example: x 1 2 3 y 5.1 5.9 6.3 -> Solution i 1 2 3 sum xi 1 2 3 6 yi 5.1 5.9 6.3 17.3 xi2 1 4 9 14 xi yi 5.1 11.8 18.9 35.8 Equations: 3𝑎𝑎 + 6𝑏𝑏 = 17.3 6𝑎𝑎 + 14𝑏𝑏 = 35.8 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆: 𝑎𝑎 = 4.5667𝑏𝑏 = 0.60 10  Quantification of Error of Linear Regression The coefficient of determination (denoted by r2) is a key output of regression analysis.  An r2 of 0 means that the dependent variable cannot be predicted from the independent variable.  An r2 of 1 means the dependent variable can be predicted without error from the independent variable.  An r2 between 0 and 1 indicates the extent to which the dependent variable is predictable. The total sum of the squares of the residuals between the data points and the mean The total sum of the squares of the residuals between the data points and the mean An alternative formulation for r that is more convenient for computer implementation is 15  Multiple Linear Regression Example : t 0 1 2 3 x 0.1 0.4 0.2 0.2 Given the following data: y 3 2 1 2 Determine a function of two variables: f(x,t) = a + b x + c t That best fits the data with the least sum of the square of errors. 16  t 0 1 2 3 x 0.1 0.4 0.2 0.2 y 3 2 1 2 Construct Φ , the sum of the square of the error and derive the necessary conditions by equating the partial derivatives with respect to the unknown parameters to zero, then solve the equations. 17 17  n f ( x, t ) = a + bx + ct , Φ ( a , b, c ) = ∑ (a + bxi + cti − yi )2 i =1 Necessary conditions : n ∂Φ ( a , b, c ) = 2∑ (a + bxi + cti − yi ) = 0 ∂a i =1 n ∂Φ ( a , b, c ) = 2∑ (a + bxi + cti − yi ) xi = 0 ∂b i =1 n ∂Φ ( a , b, c ) = 2∑ (a + bxi + cti − yi ) ti = 0 ∂c i =1 18 System of Equations n n n a n + b ∑ xi + c ∑ ti = ∑ y i i =1 i =1 i =1 n n n n a ∑ xi + b ∑ ( x i ) + c ∑ ( x i t i ) = ∑ ( x i y i ) 2 i =1 i =1 i =1 i =1 n n n n a ∑ ti + b∑ ( xi ti ) + c ∑ (ti ) 2 = ∑ (ti yi ) i =1 i =1 i =1 i =1 19  Solution: Multiple Linear Regression i 1 2 3 4 Sum ti 0 1 2 3 6 xi 0.1 0.4 0.2 0.2 0.9 yi 3 2 1 2 8 xi2 0.01 0.16 0.04 0.04 0.25 xi ti 0 0.4 0.4 0.6 1.4 xi yi 0.3 0.8 0.2 0.4 1.7 ti2 0 1 4 9 14 ti yi 0 2 2 6 10 24  System of Equations Solving the System of Equations 25  Polynomial Regression The least squares method can be extended to fit the data to a higher-order polynomial 2 2 2 f ( x ) = a + bx + cx , ei = ( f ( x ) − yi ) n Minimize Φ ( a , b, c ) = ∑ ( a + bxi + cxi2 − yi ) 2 i =1 Necessary conditions : ∂Φ ( a , b, c ) ∂Φ ( a , b, c ) ∂Φ ( a , b, c ) = 0, = 0, =0 ∂a ∂b ∂c 26  Equations for Quadratic Regression n Minimize Φ ( a, b, c ) = ∑ ( a + bxi + cxi2 − yi ) 2 i =1 n ∂Φ ( a, b, c ) ∂a ( ) = 2∑ a + bxi + cxi2 − yi = 0 i =1 n ∂Φ ( a, b, c ) ∂b ( ) = 2∑ a + bxi + cxi2 − yi xi = 0 i =1 n ∂Φ ( a, b, c ) ∂c ( ) = 2∑ a + bxi + cxi2 − yi xi2 = 0 i =1 27  Normal Equations n n n a n + b ∑ xi + c ∑ xi2 = ∑ yi i =1 i =1 i =1 n n n n a ∑ xi + b ∑ 2 xi +c∑ 3 xi = ∑ xi y i i =1 i =1 i =1 i =1 n n n n a ∑ xi2 + b ∑ xi3 + c ∑ xi4 = ∑ xi2 yi i =1 i =1 i =1 i =1 28  Example 3: Polynomial Regression Fit a second-order polynomial to the following data xi 0 1 2 3 4 5 ∑=15 yi 2.1 7.7 13.6 27.2 40.9 61.1 ∑=152.6 xi2 0 1 4 9 16 25 ∑=55 xi3 0 1 8 27 64 125 225 xi4 0 1 16 81 256 625 ∑=979 xi y i 0 7.7 27.2 81.6 163.6 305.5 ∑=585.6 xi2 yi 0 7.7 54.4 244.8 654.4 1527.5 ∑=2488.8 29  System of Equations Solving the System of Equations 30  General Linear Fitting with Nonlinear Functions xi 0.24 0.65 0.95 1.24 1.73 2.01 2.23 2.52 yi 0.23 -0.23 -1.1 -0.45 0.27 0.1 -0.29 0.24 It is required to find a function of the form : f ( x ) = a ln( x ) + b cos( x ) + c e x to fit the data. n Φ ( a , b, c ) = ∑ ( f ( xi ) − yi ) 2 i =1 31  n Φ ( a , b, c ) = ∑ ( a ln( xi ) + b cos( xi ) + c e xi − yi ) 2 i =1 Necessary condition for the minimum : ∂ Φ ( a , b, c )  = 0 ∂a  ∂ Φ ( a , b, c )  = 0 ⇒ Normal Equations ∂b  ∂ Φ ( a, b, c )  = 0 ∂c  32  Normal Equations n n n n a ∑ (ln xi ) + b∑ (ln xi )(cos xi ) + c ∑ (ln xi )( e ) = ∑ yi (ln xi ) 2 xi i =1 i =1 i =1 i =1 n n n n a ∑ (ln xi )(cos xi ) + b∑ (cos xi ) + c ∑ (cos xi )( e ) = ∑ yi (cos xi ) 2 xi i =1 i =1 i =1 i =1 n n n n a ∑ (ln xi )( e ) + b∑ (cos xi )( e ) + c ∑ ( e ) = ∑ yi ( e xi ) xi xi xi 2 i =1 i =1 i =1 i =1 Evaluate the sums and solve the normal equations. 33  Example 4: Evaluating Sums xi 0.24 0.65 0.95 1.24 1.73 2.01 2.23 2.52 ∑=11.57 yi 0.23 -0.23 -1.1 -0.45 0.27 0.1 -0.29 0.24 ∑=-1.23 (ln xi)2 2.036 0.1856 0.0026 0.0463 0.3004 0.4874 0.6432 0.8543 ∑=4.556 ln(xi) cos(xi) -1.386 -0.3429 -0.0298 0.0699 -0.0869 -0.2969 -0.4912 -0.7514 ∑=-3.316 ln(xi) * exi -1.814 -0.8252 -0.1326 0.7433 3.0918 5.2104 7.4585 11.487 ∑=25.219 yi * ln(xi) -0.328 0.0991 0.0564 -0.0968 0.1480 0.0698 -0.2326 0.2218 ∑=-0.0625 cos(xi)2 0.943 0.6337 0.3384 0.1055 0.0251 0.1808 0.3751 0.6609 ∑=3.26307 cos(xi) * exi 1.235 1.5249 1.5041 1.1224 -0.8942 -3.1735 -5.696 -10.104 ∑=-14.481 yi*cos(xi) 0.223 -0.1831 -0.6399 -0.1462 -0.0428 -0.0425 0.1776 -0.1951 ∑=-0.8485 (exi)2 1.616 3.6693 6.6859 11.941 31.817 55.701 86.488 154.47 ∑=352.39 yi * exi 0.2924 -0.4406 -2.844 -1.555 1.523 0.7463 -2.697 2.9829 ∑=-1.9923 34  System of Equations 4.55643 a − 3.31547 b + 25.2192 c = −0.062486 − 3.31547 a + 3.26307 b − 14.4815 c = −0.848514 25.2192 a − 14.4815 b + 352.388 c = −1.992283 Solving the above equations : a = − 0.88815, b = − 1.1074 , c = 0.012398 Therefore, f ( x ) = −0.88815 ln( x ) − 1.1074 cos( x ) + 0.012398 e x 35  Example 5 xi 1 2 3 Given: yi 2.4 5 9 Find a function f(x) = ae bx that best fits the data. ( ) n 2 Φ ( a, b) = ∑ ae bxi − yi i =1 Normal Equations are obtained using : ( ) n ∂Φ = 2∑ ae bxi − yi e bxi = 0 ∂a i =1 Difficult to Solve ( ) n ∂Φ = 2∑ ae bxi − yi a xi e bxi = 0 ∂b i =1 36 Non-Linear Regression: Linearization Method 37 38  Example 6 Find a function f(x) = ae bx that best fits the data. Define g ( x ) = ln( f ( x )) = ln( a ) + b x Define zi = ln( yi ) = ln( a ) + bxi Let α = ln( a ) and zi = ln( yi ) ( ) n 2 Instead of minimizing : Φ ( a, b) = ∑ ae bxi − yi i =1 n Minimize : Φ (α , b) = ∑ (α + bxi − zi )2 ( Easier to solve) i =1 39  n Φ (α , b) = ∑ (α + b xi − zi )2 i =1 Normal Equations are obtained using : n ∂Φ = 2∑ (α + b xi − zi ) = 0 ∂α i =1 n ∂Φ = 2∑ (α + b xi − zi ) xi = 0 ∂b i =1 n n n n n α n + b ∑ xi = ∑ zi and α ∑ xi + b∑ xi2 = ∑ ( xi zi ) i =1 i =1 i =1 i =1 i =1 40  xi 1 2 3 ∑=6 yi 2.4 5 9 zi=ln(yi) 0.875469 1.609438 2.197225 ∑=4.68213 xi2 1 4 9 ∑=14 xi z i 0.875469 3.218876 6.591674 ∑=10.6860 Equations : α = ln( a ), a = eα 3 α + 6 b = 4.68213 a = e 0.23897 = 1.26994 6 α + 14 b = 10.686 f ( x ) = ae bx = 1.26994 e 0.66087 x Solving Equations : α = 0.23897, b = 0.66087 41