Stereo Vision and Stereopsis

• Stereoscopic vision uses the principles of computational vision.
• Stereo vision topics include sections 19.6-19.7 and chapter 7 Stereopsis.
• Stereopsis can be visually referenced via Wikipedia.

Stereopsis

• Stereopsis is the perception of depth and 3D structure via visual input from both eyes.
• Stereoscopic perception's computer implementation has applications in visual robot navigation, aerial reconnaissance, and Cartography.
• Stereoscopic perception implementation is also used in 3D sensing/reconstruction, and industrial inspection.

NASA Curiosity Rover

• NASA's Curiosity Rover uses stereo vision for Mars exploration.
• The rover has 23 cameras total.
• 9 cameras are for engineering purposes.
• 7 cameras are for science purposes.
• 7 cameras are for entry, descent and landing purposes.

Stereo Vision

• Structure and depth are inherently ambiguous from single views.
• The goal is to determine where points P1 and P2 actually are in a 3D.
• This is resolved in stereo vision.
• Depth is inferred from different clues in an image.
• Perspective can give a notion of depth with converging lines.
• Shallow focus/blur cues depth.
• Size and scale provide depth information; smaller objects are farther away.
• Lighting/shading can give a sense of depth; harsher shadows with fast falloff create this illusion.
• Objects are closer the more they occlude things that are further.

Basic Stereo Geometry

• To determine actual location, use the different views from cameras.
• Estimating depth from stereo vision includes image point correspondence, and information on camera pose (calibration).

Simple Stereo System Geometry

• Parallel optical axis and known calibrated camera parameters are assumed.
• Baseline is the line connecting optical centres (B), and focal length is f.
• Point P is at distance Z in the camera coordinate systems (virtual images).
• The assumption of similar triangles is used for calculations for the stereo vision geometry.
• The formula for the disparity calculation is Z is equal to focal length multiplied by baseline, and divided by XL - XR.
• Disparity is equal to the difference between XL - XR.
• 0 depth means infinite disparity.

Disparity Map

• A ground truth disparity map and computed disparity map are used.

Epipolar Geometry

• The baseline is a line joining the cameras centres.
• The epipolar plane contains the baseline and a world point P.
• Epipolar lines (pairs) are the intersection of the epipolar plane with the image plane, potentially outside the image.
• An epipole is a point of intersection of the baseline with the image plane.
• Geometry of two views constraints where the corresponding pixel for some image point in the first view must occur in the second view.
• Epipolar constraint: with a calibrated stereo rig, a point p is constrained to lie on the associated epipolar line l'.

Converging Cameras

• Epipolar constraint reduces the correspondence problem to a 1D search along epipolar line for converging cameras.
• Two images with superimposed corresponding points have epipolar lines in white which can be used to analyse.

Parallel Cameras

• For parallel image planes, the baseline's intersection with the image plane is at infinity.
• The epipoles are at infinity and epipolar lines are parallel.
• An epipolar geometry is used for motion parallel to the image plane.
• Superimposed corresponding points and their epipolar lines (in white) can be used to analyse a pair of parallel images.

Image Rectification

• Stereo algorithm calculations are simplified when images of interest are rectified – replaced by two equivalent pictures with a common image plane parallel to the baseline joining the two optical centres.
• The two image planes II and II' are reprojected onto a common plane parallel to the baseline.
• Epipolar lines l and l' associated with points p and p in the two pictures map onto a common scanline also parallel to/passing through reprojected points.
• Simplified parallel planes are needed, since they are assumed to be co-planar with the same focal lengths.
• The epipolar lines are horizontal and at the same y location in the image.

Stereo Correspondence

• The epipolar constraint is a hard constraint.
• Additional soft constraints include similarity where corresponding pixels should look alike.
• Also, there should be uniqueness (no more than one match).
• Ordering makes pixels appear in the same order in both images.
• Limited disparity gradient means that depth doesn't change rapidly.
• To find matches in an image pair, it's assumed that scene points are visible in views, and that image regions for the matches are similar in appearance.

Dense Correspondence Search

• Each pixel/window in the left image must be compared with every pixel/window on same epipolar line in right image.
• The pick position must have a minimum match cost which include sum of squared differences, and normalized correlation, for different intensity images.

Occlusion Problem

• On the left, the order of feature points along the two (oriented) epipolar lines is the same.
• On the right, surface points aren't visible in one image and the image points' order are not the same in the two pictures.

Dynamic Programming Formulation

• Instead of individual correspondences to estimate disparities, optimise correspondence assignments joinly.
• This can be done via scanline, and through a full 2D grid (graph cuts).

Stereo Correspondence Challenges

• What defines good stereo correspondence includes having match quality, and smoothness.
• Challenges with stereo correspondence includes low contrast and textureless image regions.
• Other challenges consist of occlusions, violations of brightness constancy, and really large baselines (foreshortening/appearance change).
• Final challenges include camera calibration errors.

Disparity map - MATLAB Example

• To load a rectified stereo pair image, use I1=imread('rectified_left.png') and I2=imread('rectified_right.png').
• The Anaglyph function creates the stereo anaglyph of the image and allows it to be used with stereo glasses, and can be called by A=stereoAnaglyph(I1,I2).
• Convert rectified input color images to grayscale images using: J1 = rgb2gray(I1) and J2 = rgb2gray(I2).
• Specify disparity range and the uniqueness threshold to compute with disparitySGM function.
• The disparityMap = disparitySGM(J1,J2, 'DisparityRange',disparityRange, 'UniquenessThreshold',20) command calculates a disparity map through semi-global matching.
• The display disparity map sets display range to same value as range: imshow(disparityMap,disparityRange).

Project 2

• Course project consist of 7 topics, and use MATLAB examples, replicate the example and apply the same strategy to a case.
• Project due on April 22nd.
• Project submission through Fenix.

Next Class

• Next class will discuss projective geometry.