Multivariable Second Derivative Test

• Chapter 26 applies the quadratic approximation to functions from R" to R near critical points to create a multivariable second derivative test.
• This test determines if the graph of f inside R^(n+1) near a point is bowl-shaped (local maximum), upward (local minimum), or saddle-like.
• The Hessian matrix (Hf)(a) and its associated quadratic form are essential tools.

Quadratic Form

• Defined as q(x) = ∑ bᵢⱼ xᵢxⱼ (where the sum of i is from 1 to n, and the sum of i and j is constrained by 1 <= i < j <= n)
• bij = fxixj(a), connecting quadratic forms with symmetric matrices based on the link in (20.3.1) and Example 20.3.12
• Understanding the geometry of level sets q(x1, . . ., xn) = c is a key question.
• Eigenvectors are used to systematically understand level sets for any n, building on Section 25.4 results for n = 2
• Analyzing the quadratic form associated with the Hessian at a critical point a of f : R" → R determines whether a is a local maximum, local minimum, or saddle point.
• This approach recovers the second derivative test from single-variable calculus when n = 1.

Chapter Objectives

• Identify local maxima, minima, and saddle points from contour plots when n = 2
• Determine if a critical point a ∈ R" is a local maximum, minimum, or saddle point when (Hf)(a) is a diagonal matrix
• Use eigenvalues to determine the definiteness of the Hessian and use that information to identify local maxima, minima, and saddle points

Definiteness and Saddle Points

• The quadratic approximation to f near a point a ∈ R" is given by f(a + h) ≈ f(a) + (∇f)(a) · h + (1/2)hᵀ((Hf)(a))h for h ∈ R" near 0
• For n = 1, this simplifies to the Taylor approximation f(a + h) ≈ f(a) + f'(a)h + (1/2)f"(a)h²
• At a critical point a, where (∇f)(a) = 0, the quadratic approximation becomes f(a + h) ≈ f(a) + (1/2)hᵀ((Hf)(a))h

Single-Variable Calculus

• If f"(a) > 0 then (1/2)f"(a)h² is positive for all h ≠ 0, indicating a local minimum at a
• Conversely, if f"(a) < 0 then (1/2)f"(a)h² is negative for all h ≠ 0, indicating a local maximum at a
• To determine if f(a) is a local maximum/minimum, we examine if q(Hf)(a)(h) = hᵀ((Hf)(a))h is consistently negative/positive for small h near 0
• Definition 24.2.2 asks: is the Hessian matrix (Hf)(a) negative-definite, positive-definite, or something else?
• Variation in values of an n-variable quadratic form near 0 becomes more complex when n > 1.
• For n = 2, contour plots near (0,0) illustrate this complexity, with level sets forming hyperbolas, ellipses, or pairs of parallel lines
• For an n × n symmetric matrix A, qa(x) = xᵀAx ∈ R is a function of x ∈ R". Examples include:
• n = 2: ax² + by² + 2uxy
• n = 3: ax² + by² + cz² + 2uxy + 2vxz + 2wyz
• Off-diagonal matrix entries contribute a factor of 2 to the right side of the equations above.
• Critical points motivate interest in determining where such functions q(x) are positive or negative.
• For n > 1, there is the possibility of a saddle point, which has no analogue for n = 1.
• For a critical point a = (a1, a2) ∈ R² of f, the surface graph z = f(x, y) may have a local minimum along one line L in R² through a, and a local maximum along another line L'
• This results in a "saddle" appearance of the surface graph z = f(x, y) near (a1, a2, f(a1, a2)) ∈ R³

Saddle Shape

• In Example 10.2.8 with f(x, y) = x² - y², a "saddle" appears over (0,0) ∈ R²
• This shape stems from vertical planes P and P' that cut the surface graph S through (0,0, f(0, 0)) along curves C and C' with opposite local extrema
• The absence of a local extremum can occur along a line through a saddle point
• planes through (0, 0, f(0, 0)) correspond to choosing a line l in the xy-plane through (0,0), with the curve being the graph of f(x, y) over the line l
• Many lines l exhibit local extrema at (0,0) for f(x,y) = x² - y² = , for example, local maximum, local minimum

Definition of a Saddle Point

• A critical point a ∈ Rⁿ is a saddle point of f if there are distinct lines L, L' in Rⁿ through a such that f evaluated on L has a local maximum at a, and f evaluated on L' has a local minimum at a
• L and L' can be written in parametric forms {a + tv} and {a + tv'}
• Functions g(t) = f(a + tv) and h(t) = f(a + tv') indicate the behavior of f on points of L and L'
• The saddle-point condition dictates that g has local maximum at t = 0 and h has a local minimum at t = 0
• Contour plots near saddle points for n = 2 show that among pairs of lines through a with opposite local extrema, a special pair exists: the "lines of symmetry"
• The lines of symmetry reflect the contour plot near a and are given by {a + tw} and {a + tw'}, where w and w' are orthogonal eigenvectors of (Hf)(a)

Second Derivative Test

• For a critical point a ∈ Rⁿ of f, and small h, we have f(a + h) ≈ f(a) + (1/2)hᵀ((Hf)(a))h.
• If q(Hf)(a)(h) > 0 then f(a + h) > f(a), whereas if q(Hf)(a)(h) < 0 then f(a + h) < f(a).
• This leads to the Second Derivative Test, Version I for f : R" → R at a critical point a ∈ R":
• If (Hf)(a) is positive-definite then a is a local minimum of f
• If (Hf)(a) is negative-definite then a is a local maximum of f
• If (Hf)(a) is indefinite then a is a saddle point of f
• If the Hessian (Hf)(a) is positive-semidefinite or negative-semidefinite but not definite, more information is needed.
• To find definiteness properties of the Hessian at a, we will use determinates and eigenvalues which is challenging
• For n = 1, we apply above since hᵀ(Hf)(a)h = f"(a)h²

Convexity

• (Hf)(a) is positive-definite for all a ∈ R" exactly when f is "strictly convex": f((1-t)v + tw) < (1-t)f(v) + tf(w) for all v ≠ w and 0 < t < 1
• If the symmetric n × n matrix (Hf)(a) is diagonal, definiteness properties can be determined by inspecting signs of diagonal entries.

Diagonal entries

• Consider H with diagonal entries 11, 3, 4 the matrix.
• The diagonal entries are the coefficients in the associated quadratic form which is positive when ≠ 0
• If all aⱼ's are positive, H is positive-definite (aⱼ > 0). H is indefinite if one aᵢ's is positive and another is negative
• It's easy to check definiteness/indefiniteness of a diagonal n×n matrix D by inspection - evaluate qD(x) on coordinate axes
• Suppose H is a 2 × 2 "anti-diagonal" matrix. This is indefinite because is indefinite because 10xy has positive and negative values

Gram Matrix

• If M is any m × n matrix, the n × n matrix B = MᵀM (Gram matrix) is symmetric (Theorem 20.3.8)
• For any v ∈ Rⁿ, qB(v) = vᵀ(MᵀM)v = ||Mv||² ≥ 0. Hence, B is positive-semidefinite
• If the null space of M is {0} then B is positive-definite

Gram Matrices

• Every positive-semidefinite symmetric n × n matrix B is MᵀM for an n x n matrix M
• QR-decomposition (possibly non-invertible M) gives QR where Q is an n x k matrix orthogonal columns gives: B = MᵀM = RᵀR
• This special LU-decomposition (L = Uᵀ, with U = R positive diagonal) is the Cholesky decomposition

Second Derivative Test

• Analyze whether an extremum at a point is local
• restrictions of h to the coordinate axes have a local minimum at the origin of the axes
• The Hessian matrix for h is diagonal everywhere
• By Theorem 26.1.5, the critical point is a local minimum for h with a positive definite point

Hessian Encoding Geometry

• the real significance is for applications beyond the 2-variable case
• Find all the critical points for a quadratic function like 3x²y + 2y³ - xy and classify each as a local maximum, local minimum, or a saddle point
• The Hessian is diagonal and anti-diagonal where the matrix is definite, positive definite, or negative definite

Contour plots

• At each critical point the definiteness matches the contour plot

• Can look through a contour plot or use linear algebra to see the critical points to determine optimization and thermodynamics

• The local maximum is at a point where all partials vanish, making the result definite negative

• Find if is a local maximum, a local minimum, a saddle point, or perhaps something worse and analyze Hessian

• constant λ disappears and we obtain

Hessian

• The Hessian is not diagonal so not easily definitive through the eigenvalues. In other words, the spectrum is not diagonal
• Not always a saddle point due to symmetry
• In two variables, this can appear as a diagram like tacnode

Eigenvectors and Hessians

• Now use Theorem 26.3.4 to understand the behavior of some 2-variable functions near critical points, and also apply the eigenvalue method to a 3-variable function with non-diagonal Hessian at two critical points
• If a point is stable and positive definite, it is a local minimum and vice versa

Computing Eigenvalues

• Eigenvalues can be computed by knowing the trace, or if it is already a saddle-point due to symmetry on a 2D or 3D level
• One seeks a local minimum, and must determine what form the Hessian takes, whether diagonal or zeroed
• If not zeroed the same is applied
• A positive definite shows that the approximate critical point we found is a local has determinant

Hessian at a general point

• The eigenvalues are negative if the points are negative and and negative-definite.
• The point attains a global maximum in the interior near the critical level

Critical Points

• Find the critical points and the behavior of each
• the vanishing of fy forces x = y
• The determinant computes to either positive or negative as a definite equation. It also depends on the common sign of eigenvalues
• The method helps prove what must be positive

Relationships in this Chapter

• The relationship between the behavior of a function f : R" → R near a critical point a and the eigenvalues and eigenvectors of the Hessian (Hf)(a), especially when all eigenvalues are nonzero
• A Hessian having only one nonzero eigenvalue with n = 2,3 the computer vision technique underlies ridge detection
• Useful in areas such as: facial recognition with wrinkles, detecting blood vessels in brain imaging, and digital humanities to reconstruct