Understanding Camera Models and Geometry

Camera Basics

• A camera captures light from a scene and records an image using a sensor.
• The sensor can be digital, using CMOS or CCD technology, or analog.
• Cameras are essential for capturing visual details.

Camera Geometry

• A camera captures a real 3D scene, converts it into a 2D format, which leads to the loss of one dimension.
• Computer vision helps recover missing information from the 2D images.
• Computer vision provides high-level image understanding and restores 3D details from 2D conversions.

Camera Models

• A camera model represents the projection of 3D information into a 2D scene.
• Camera models aid in calibration, computer vision tasks, and reconstruction of scenes.
• Models vary based on lens types and projection methods with different applications requiring specific camera models.
• Understanding these models enhances overall image processing tasks; models are essential for accurate vision

Types of Camera Models

• Pinhole Model: Used for basic image projection.
• Thin Lens Model: Suited for real-world photography.
• Affine Model: Used for flat imaging applications.
• Projective Model: For 3D reconstruction.
• Fish-eye Model: Used when wide-angle imaging is needed.
• Omnidirectional Model: Needed for robotics and self-driving cars.

Comparison of Camera Models

• Pinhole: No Lens, has Depth Info, used in CV, AI, no distortion; needs high light.
• Thin Lens: Has Lens & Depth Info, used in Photography, Autofocus/Zoom, Lens distortion.
• Affine: Has Lens, No Depth Info, used in Satellie, Scanner, Fast Processing, No perspective.
• Projective: Has Lens & Depth Info, used in 3D Vision, AI, Realistic, Complex Math
• Fish-Eye : Has Lens & Depth Info, used in VR, 360 camera, Ultrawide view, Image Warping.
• Omnidirectional: Has Lens & Depth info, used in Robotics/Self-Driving, 360 Capture, Complex calibration.

Pinhole Camera

• It represents a 3D scene into a 2D image plane.
• A pinhole camera operates without lenses, relying on a small aperture that allows light rays to pass through.
• A pinhole camera forms an inverted image on the opposite side of the aperture.

Pinhole Camera: How it works

• The pinhole camera consists of a small hole aperture that lets light in.
• It has a light-tight box that prevents other light from entering.
• It has either a screen or film where the image is projected.

Pinhole Projection Mechanism

• When light from a 3D object passes through the pinhole, the image forms upside down on the image plane.
• A smaller pinhole produces a sharper image but makes it darker.

Pinhole Camera Working Explained

• "f" represents the distance between the image and the screen.
• "z" represents the object's distance from the screen.
• "(x, y, z)" represents the real 3D world coordinates.
• "(u, v)" represents the 2D image coordinates.

Pinhole Geometry

• Both triangles in a pinhole camera share the same angle ratios.
• Image size (u, v) is directly proportional to "f" (focal length); larger focal lengths increase image size.
• Image size (u, v) is inversely proportional to "z" .

Pinhole Camera Equation for Image Formation

• x' = f * (X / Z)
• y' = f * (Y / Z)
• (X, Y, Z) = 3D coordinates of the object
• (x', y') = 2D coordinates on the image plane
• f = Focal length (distance from pinhole to image plane)

Pinhole Camera Explanation

• The farther an object is, the smaller it appears.
• The closer an object is, the larger its projection.

Pinhole Camera: Advantages

• These cameras have no lens distortion.
• Simple and cheap to construct.
• Provides a large depth of field, keeping everything in focus.

Pinhole Camera: Disadvantages

• Low brightness due to the small aperture.
• Requires long exposure times to capture images.
• Limited sharpness because diffraction blurs details.

Pinhole Camera Example

• An object 5 cm tall is placed 50 cm away from a pinhole camera with a focal length of 10 cm and the image height is 1 cm, and it is inverted.

Camera with Lenses

• The camera with lenses is a real-world model where a lens system is used to focus light onto an image sensor.
• The model overcomes the limitations of a simple pinhole camera by allowing more light to enter while maintaining sharp focus

Camera with Lenses : How it works

• Light from a 3D scene enters the camera lens.
• The lens focuses the light onto the image sensor (CCD/CMOS).
• The sensor then converts the light into an electronic image which is captured and stored digitally.

Camera with Lenses Equation

• 1/f = 1/do + 1/di
• f = Focal length (distance where light converges).
• do = Object distance (distance from object to lens).
• di = Image distance (distance from lens to sensor).
• Focal Length controls zoom & perspective distortion.

Camera Lenses

• Convex Lens (Converging Lens): Focuses light to a single point on the sensor, and is in most DSLRs, mobile cameras, and webcams.
• Concave Lens (Diverging Lens): Spreads light out, and reduces magnification, and is used in wide-angle and virtual reality lenses.
• Zoom Lens: Variable focal length, adjusts zoom level, found in CCTV cameras and DSLR lenses.
• Wide-Angle Lens: Captures a wider field of view, used in sports and landscape photography.

CCD (Charge-Coupled Device) Cameras

• CCD cameras are a type of digital imaging device that converts light into electrical signals.
• CCD cameras are used in astronomy, microscopy, medical imaging, and industrial inspection.
• Used because of high sensitivity and low noise.

CCD Cameras : How they work

• Light Capture: Photons from a scene hit the CCD sensor.
• Electron Generation: The sensor converts light into electrons.
• Charge Storage: Each pixel stores charge proportional to light intensity.
• Charge Transfer: Charges move across the chip in a controlled manner.
• Readout and Conversion: The charge is converted to a digital format.

Advantages of CCD Cameras

• Provides high image quality and high sensitivity, working well in low-light conditions.
• Produces low noise, generating clearer images
• It has a uniform pixel response, with no variation in brightness or color across pixels.
• Provides good dynamic range which captures both bright and dark areas accurately.

Disadvantages of CCD Cameras

• Has high Power Consumption, which requires more power than CMOS sensors.
• Expensive Manufacturing: More costly to produce than CMOS cameras.
• Slower Readout Speed: Takes longer to capture and process images.
• Blooming Effect: Has excess charge that can spill over to neighboring pixels.

Lens Distortion

• Deviation from a perfect projection of a scene occurs due to imperfections in a camera lens.
• Straight lines in the real world appear curved or misshaped in an image.
• Different parts of the lens bend light unevenly

General Projective Cameras

• The projective camera models the perspective projection of a 3D scene onto a 2D image plane by the camera projection matrix P=K [R|t].
• K defines the focal length, principal point, and skew.
• [R | t] defines the camera position and orientation in 3D space.

General Projective Cameras: Homogenous Coordinates

• 3D world points (X, Y, Z, 1) are mapped to 2D image points (x, y, 1) via projective transformation.
• Given a 3D point (X,Y,Z)=(10,5,20), its homogeneous form is (10,5,20,1).
• General Projective Cameras are used for 3D vision, AR, robotics, and mapping.

Intrinsic Parameters (K)

• Intrinsic parameters define internal camera characteristics that convert real-world distances into pixel coordinates.
• This includes the focal length, principal point, and pixel scaling.

Intrinsic Camera Matrix (K)

• K = [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
• fx, fy represents the focal length in pixels (scaling factor for x and y).
• cx, cy is the principal point (center of the image sensor).
• The last row [0, 0, 1] ensures homogeneous coordinates.
• Used for converting 3D camera coordinates into 2D image coordinates.
• Example: Given K Matrix K = [[800, 0, 320], [0, 800, 240], [0, 0, 1]]
• 800 px is the focal length.
• (320, 240) px is the center of the image.

Extrinsic Parameters ([R | t])

• Extrinsic parameters describe the position and orientation of the camera in the world coordinate system, and consist of:
• Rotation Matrix (R): Describes the camera's orientation.
• Translation Vector (t): Describes the camera's position in the world.
• Extrinsic Camera Matrix ([R | t]) = [[R3x3, t3x1], [0, 1]]

Extrinsic Parameters: Example

• Given a Rotation Matrix (R) where the camera does not rotate(Identity Matrix): R = [[1, 0, 0], [0, 1, 0], [0, 0, 1]] applies no rotation to the camera.
• Given a Translation Vector (t) where the Camera is shifted along the Z-axis: t = [[0], [0], [-50]], the camera is 50 units behind the world origin.

Complete Projection Formula

• [x, y, w] = K[R|t] * [X,Y,Z,1]
• Where:
  • Intrinsics (K) are camera properties.
  • Extrinsics ([R | t]) are the camera pose in the world.
• Finally, we compute the 2D image coordinates: x' = x/w , y' = y/w

Step-by-Step Calculation

• Construct an Extrinsic Matrix [R | t].
• Multiply with the 3D World Point (Homogeneous Coordinates).
• Apply the Intrinsic Matrix (K).
• Converts to Pixel Coordinates (x', y').

Affine Cameras

• It is a simplified version of a projective camera where perspective effects are ignored.
• Parallel lines in 3D remain parallel in 2D.
• It represents a 3D point (X,Y,Z) as a 2D image point (x,y) using the equation: [x, y] = A[X, Y, Z] + t

Affine Camera Coordinates

• "A" is a 2x3 affine transformation matrix (rotation, scaling, shear).
• "t" is a 2x1 translation vector (shifting the image).

Application

• It can be written in homogenous coordinates: [x, y,1] = [A t; 0 1] [X, Y, Z,1]

Weak perspective projection

• This is accurate when the object is small and distant and is most useful for recognition.

Affine Projection example

• Multiply the 2x4 affine projection matrix A with the 4x1 homogeneous world point to get the project 2D image coordinates.

Affine Cameras : Key Properties & Application

• Provides no Perspective Distortion where Parallel lines remain parallel after transformation.
• It provides Linear Projection that Preserves ratios and angles in small regions.
• It is used in projection and planar motion tracking, and face alignment.
• Works well when the camera is far from the object

Camera Calibration

• It estimates the intrinsic (K) and extrinsic parameters ([R | t]) to correct distortions of the camera.
• It estimates the parameters of lens and image sensor of an image or video to correct for lens distortion.
• Is used in machine vision to detect and measure objects like robotics, navigation systems, 3-D scene reconstruction.

Summary of Camera Models

• Pinhole Camera uses Perspective Projection, and is used for 3D Reconstruction
• Affine Camera uses Weak Perspective, and is used for Object Recognition.
• Projective Camera uses Full Perspective Model, and is used for AR/VR Applications.
• Fisheye Camera uses Wide-Angle Distorted, and is used for Surveillance/VR

Projective Geometry

• Projective Geometry describes how 2D points transform under perspective projection.
• It extends Euclidean geometry by introducing homogeneous coordinates.
• Allows for operations such as scaling, translation, rotation, and perspective transformation.

Planar Geometry & Projective Spaces

• Euclidean Geometry: X=[x,y] , regular 2D geometry where we represent a point as (x,y).
• Projective Geometry: X = Kp [x,y,1] = [Kpx, Kpy, Kp], extends Euclidean geometry, handles perspective transformations

Projective Geometry Homogenous Coordinates

• In projective geometry, every 2D point (x, y) is written in homogeneous coordinates: (x, y) → (x, y, w)
• "w" is a scaling factor that requires the user to convert back when not being used.
• To convert back, divide the x,y by the scaling factor: (x, y, w) → (x/w, y/w)

Homogeneous Coordinates

• An extra dimension will need to be added to the bottom of the vector and must have a "1" for entry.
• The entire new vector is then multiplied by an arbitrary scaling factor kp.

Homogeneous coordinates original representation

• To return to the original x, y representation, divide the first two vector entries by the third entry, which is always equal to the scaling factor kp.

Homogenous Coordinate Conversion Example

• To convert the Euclidian Point (4,2) to homogenous coordinates it becomes ((4,2) --> (4,2,1)).
• Now to convert back it must be ((4,2,2) --> (4/2 , 2/2) = (2,1)).

Homography

• A homography is a special matrix that transforms one plane to another.
• Preserves straight lines and it is represented as a 3×3 matrix.
• Homography preserves collinearity and cross ratio.

Homography Equation

• Mathematically, the homography equation is: [[x',y',w'] = H [[x,y,w]]

Homography Explained

• (x,y,w) are the homogeneous coordinates, and (x',y',w') are the transformed coordinates.
• H (homography) is the homography matrix, to convert back divide [x/w,y/w]

Properties of Homography

• It can map straight lines to straight lines, and preserve "Collinearity"(Points that lie on a line remain on a line after transformation).
• Can handle perspective distortions like a tilted book page appearing rectangular.
• You need at least 4 points to Compute.

Applications of Homography

• Camera has Images Stitching (Panoramic Photos) that takes two overlapping images and aligns them.
• Is mostly used in Google Photos and Photoshop to provide Perspective Correction
• Can be used to correct tilted images, such as scanning documents.