Computer Vision - Optical Flow

Optical Flow

• Optical flow is the pattern of motion of objects, surfaces, and edges in a visual scene
• It is used to understand the motion of the camera and objects in a scene

Scene Interpretation

• Given a video sequence, we can find the motion of the camera and objects by:
  • Recovering camera ego-motion
  • Motion segmentation
  • Structure from motion

Applications

• Optical flow has applications in:
  • Recovering camera ego-motion
  • Motion segmentation
  • Structure from motion
  • Multi-body segmentation
  • Recognizing the type of motion (e.g., walking, running, etc.)

Motion Field and Optical Flow

• Motion field: the real-world 3D motion
• Optical flow field: the projection of the motion field onto a 2D image

Examples of Motion Fields

• Forward motion
• Rotation
• Horizontal translation
• Closer objects appear to move faster

When Does It Break?

• The optical flow field can break when:
  • The screen is stationary, but objects generate motion
  • Non-rigid objects change shape
  • The light source changes

Methods for Determining Optical Flow

• Phase correlation methods
• Block-based methods
• Differential methods, including:
  • Lucas-Kanade method
  • Horn-Schunck method
  • Buxton-Buxton method
  • Black-Jepson method

Estimating Optical Flow

• Assume the image intensity is constant
• Use the brightness constancy equation: I(x, y, t) = I(x + dx, y + dy, t + dt)
• First-order Taylor expansion: I(x, y, t) + Ix * dx + Iy * dy + It * dt = 0

Brightness Constancy Equation

• Simplify the notation: Ix * u + Iy * v = -It

Problem I: One Equation, Two Unknowns

• The brightness constancy equation has one equation and two unknowns (u and v)

Problem II: The Aperture Problem

• For points on a line of fixed intensity, we can only recover the normal flow
• We need additional constraints to solve the aperture problem

Local Smoothness and Lucas-Kanade

• Use local information to solve the aperture problem
• Lucas-Kanade method (1984): assume constant (u, v) in a small neighborhood

Least-Squares Method

• Solve the overconstrained linear system using the least-squares method
• Minimize the sum of squares of "errors" between the right- and left-hand sides of the equations

How Does Lucas-Kanade Behave?

• Edge regions: A^T * A becomes singular
• Homogeneous regions: A^T * A is close to zero
• Textured regions: two high eigenvalues

Other Break-Downs

• Brightness constancy is not satisfied
• A point does not move like its neighbors
• The motion is not small (Taylor expansion doesn't hold)

Multi-Scale Flow Estimation

• Use multi-scale estimation to handle large motions
• Run Lucas-Kanade at multiple scales and warp and upsample the results

Example: Motion-Based Segmentation

• Input: a video sequence
• Segmentation result: separate objects moving in different ways

Affine Motion

• For panning camera or planar surfaces, use affine motion: u = p1 + p2 * x + p3 * y, v = p4 + p5 * x + p6 * y
• Only 6 parameters to solve for, resulting in better results

This quiz covers optical flow in computer vision, understanding motion in video sequences, and scene interpretation. It involves tracking pixel movements between images and analyzing camera/object motion.