Local and Global Illumination Models

• Local illumination models consider light, surface, and camera properties.
• Inter-reflection from surrounding surfaces significantly contributes to light.

Models of Light Sources

• Distant point sources, like the sun, provide illumination from essentially one direction.
• Area sources, such as white walls, diffuser lamps, and the sky, are a type of light source.
• Ambient light serves as a substitute for explicitly modeling inter-reflections.
• Global illumination models account for inter-reflections within a modeled scene.

Pixel Brightness Factors

• A pixel’s brightness hinges on:
  • Light source characteristics (strength, direction, color).
  • Surface orientation relative to the light source and camera.
  • Surface material and albedo (surface reflectance).
  • Reflected light and shadows from surrounding surfaces.
  • Sensor gain.
• A single pixel's brightness alone reveals limited information.

Intensity Differences and Shape

• Intensity variations offer insights into shape through:
  • Changes in surface normal.
  • Texture.
  • Proximity effects.
  • Indents and bumps on the surface.
  • Grooves and creases.

Color Constancy

• Color constancy involves interpreting a surface based on its albedo or "true color," rather than the raw observed intensity.
• Humans excel at color constancy, but computers struggle to achieve comparable performance.

Color Correction

• Adjusts colors by multiplying the R, G, and B values by separate constants.
• There are several methods to determine the constants:
  • White World Assumption: Assumes the brightest pixel is white and normalizes all channels by the largest value.
  • Gray World Assumption: Assumes the average color should be gray and adjusts each channel (e.g., multiply R by avg(R)/avg((R+G+B)/3)).
  • White Balancing: Selects a reference point as the "white" or "gray" color for image correction,

Image Filtering

• Forms a new image by modifying pixel values locally using predefined rules (e.g., blurring or edge detection).
• Filtering operations typically consider neighboring pixels to compute the new value for each pixel in the output image.

Image Transformation

• Forms a new image by modifying global spatial properties of an image (e.g., rotation, scaling, or contrast adjustments).
• Transformation operations change the overall appearance or layout of the image without necessarily affecting individual pixel values directly.

Filtering

• Filters are used to extract useful information from images, such as identifying edges or contours to understand the shape of objects.
• Filters help improve image quality by removing noise or sharpening details.

Linear Filtering

• A common type of neighborhood operator.
• Works by taking a small neighborhood of pixels around each pixel, multiplying their values by filter coefficients, and adding them up to become the new pixel value.
• This operation is called correlation.
• Alternatively, rotating the filter results in convolution and impulse response function.

Separable Filtering

• A technique to optimize convolution by breaking down the kernel into two one-dimensional convolutions.
• Convolving with a 2D filter kernel requires significant computation.
• Separable filtering reduces computational load, requiring fewer operations per pixel and resulting in faster processing.

Examples of Linear Filtering

• Box Filter (Moving Average)
• Bilinear (Tent) Filter
• Gaussian Kernel
• Sobel Operator
• Corner Operator

Box Filter (Moving Average)

• Averages pixel values within a KxK window.

Bilinear (Tent) Filter

• Weights are not uniform like the box filter.
• The center pixel has a higher weighting, decreasing linearly towards the edges.
• Used to smooth an image and preserving edges and is commonly used in image resampling and scaling.

Gaussian Kernel

• Commonly used for blurring and smoothing.
• Based on the Gaussian function.
• Kernel size determines the extent of blurring, with larger kernels producing more smoothing.
• Standard deviation controls the spread of the kernel; a larger standard deviation leads to more pronounced blurring.

Smoothing and Sharpening

• Sharpening is achieved by subtracting the blurred version from the sharpened one, highlighting edges and fine details.

Band-pass Filters

• Removes both low- and high-frequency extremes.
• Typically created by smoothing with a Gaussian filter and applying first or second derivatives.
• Low frequencies in an image correspond to large uniform areas and smooth transitions.
• High frequencies correspond to rapid changes in intensity and fine details like edges, textures, or noise.

Derivatives

• Used to measure changes in intensity.
• First Derivative: Measures the rate of change of pixel intensity, highlighting edges and areas where intensity rapidly changes with a Sobel filter.
• Second Derivative: Measures the rate of change of the first derivative, captures finer details like corners and noise, and detects points where intensity changes suddenly with a Laplacian filter.
• Differences:
  • First Derivative detects edges.
  • Second Derivative detects finer details.

Laplacian of Gaussian

• Blurring an image with a Gaussian filter followed by the Laplacian operator (second derivative).
• This process is equivalent to direct convolution with a Laplacian of Gaussian filter.
• It highlights regions of rapid intensity change.

Steerable Filters

• Filters can be oriented in any direction, allowing for adjustable responses based on the image content.
• Useful for enhancing features like edges in specific orientations.
• Can be used to construct both feature descriptors and edge detectors.

Summed Area Table

• Also known as an integral image.
• Used for accelerating image processing, particularly for images with repeated convolutions with box filters.

Integral Image

• A table holding the sum of all pixel values to the left and top of a given pixel, inclusive.
• Created by iterating through the original image, where each pixel at location (i, j) contains the sum of all pixels above and to the left of (i, j), including the pixel itself.

Padding (Border Effects)

• Handling edges in image convolution can be challenging due to insufficient surrounding pixels.
• Techniques to Overcome this Issue:
  • Zero: set all pixels outside the source image to 0.
  • Constant (border color): set all pixels outside the source image to a specified border value.
  • clamp: repeat edge pixels indefinitely.
  • cyclic wrap: loop "around" the image.
  • mirror: reflect pixels across the image edge.
  • extend: extend the signal by subtracting the mirrored version of the signal from the edge pixel value.

Non-Linear Filters vs Linear Filtering

• Non-Linear Filters
  • Adjust, select, or combine pixel values based on complex relationships, resulting in non-linear output images.
  • More complex to implement and optimize.
  • Behavior isn't easily analyzed via frequency response methods.
  • Preserve sharp edges while effectively reducing or removing noise.
• Linear Filters
  • Use weighted sums of input pixels to generate output images. Simple to use and analyze with frequency responses.
  • Ineffective against certain types of noise; they tend to blur rather than remove noise.

Median Filtering

• Selects the median value from a pixel's neighborhood to filter out extreme values such as spike noises
• Is a robust alternative to averaging filters
• However, this technique is less effective at smoothing away Gaussian noise due to its dependency on a single pixel value.
• To overcome this imitation techniques such as the α- trimmed mean or weighted median can be used.

Bilateral Filtering

• Combines a Gaussian domain filter and a range filter (intensity similarities to center pixel value) to selectively smooth images while preserving edges
• Edge preservation denoising filter
• Preserves edges, unlike traditional filtering for effective smoothing of noise while maintaining the sharpness, particularly useful in color images.
• Cons: Slower than simpler linear, computational resources are required
• Enhancements: Several acceleration techniques have been developed, such as the bilateral grid and permutohedral lattice, to make bilateral filtering applicable for real-time applications.

Binary Image Processing

• Binary images only have pixel values of 0 and 1.
• Involves Morphological Operations that are used to modify the structure or shape of objects in binary images

Operations

• Dilation: Expands (Thickens) By setting a pixel to 1.
• Erosion: Reduces (Thins) Objects by having to retain all pixels at 1.
• Majority: Sets a pixel to 1 if the majority of pixels under the structuring element are 1.
• Opening: Applying erosion then dilation, creating smooth boundaries,
• Closing: Smooths Object boundaries

Distance Transforms

• A tool to calculate the shortest distance from any point nearest boundary, using either Euclidean distance or other distances.
• Applying image stitching, and fast chamfer-matching
• Areas such as object recognition, and scene interpretation

Connected Components

• Connecting pixels in an image where all adjacent components are equal
• It is crucial for applications like recognizing individual characters
• adjacent

Course Information

• DS-473 is a Computer Vision course within the Bachelor of Science in Computing, Computer Science program.
• The course aims to develop deep understanding of signal processing in frequency domain.
• The course also aims to develop proficiency in applying Fourier transforms to real-world image processing challenges.
• The course covers:
  • Fourier Transform and Filters: Analysis of signal decomposition and various filtering methods.
  • Image Classification and Segmentation: Utilization in medical imaging, photo tagging, and autonomous driving. Challenges and Traditional Approaches: Addressing issues like high variability and similarity in classes.
• Required reading includes Chapter 3, sections 3.4-3.5.3.
• Recommended reading includes Chapter 3, sections 3.1-3.7.

Thinking in Frequency

• The Fourier Transform decomposes a signal into its frequency components, to understand how filters manipulate different frequency ranges.

Filters

• Filters are designed to affect signals based on frequency.
• Low-pass filters:
  • Pass low frequencies while attenuating high frequencies.
  • Used for smoothing signals and removing high-frequency noise.
• High-pass filters:
  • Pass high frequencies and attenuate low frequencies.
  • Used in edge detection by removing low frequencies such as hum in audio.
• Band-pass filters:
  • Pass a specific range of medium frequencies, and attenuate below and above this range.
  • Used in feature extractions and texture analysis.
• A filter's effect on high, medium, and low frequencies is analyzed by passing a sinusoid of known frequency through it and observing the attenuation.
• Each filter has a frequency response, which the filter attenuates different frequencies.
• Each filter frequency response is similar to the output of a Fourier Transform.

Sinusoidal Signals

• Defined as periodic signal with a waveform as that of a sine wave
• The sine wave's parameters are:
  • Amplitude: Increases from 0 at an angle of 0° to a maximum of 1 at 90°, reaches a minimum of -1 at 270°, and returns to 0 at 360°.
  • Time period: Defined as the time taken by a periodic signal to complete one cycle; after 360°, the signal repeats. The time period is 2π.
  • Frequency: Number of oscillations in one second, mathematically defined as the reciprocal of a period.
  • Phase: Its horizontal position of a waveform in one oscillation.

Sinusoids for Analyzing Filters

• Behavior on different frequency ranges, sinusoids of known frequencies are passed through the filter and the output is observed.
• When a sinusoidal signal is convolved with a filter's impulse response, a sinusoid of the same frequency is produced with different magnitude and phase is produced.
• New magnitude A is the filter's gain or magnitude.
• The phase difference (Δ=ϕ -ϕ) is called the shift or phase.
• A large change in magnitude with a strong effect on the original sinusoid.
• Minimal change in magnitude means the filter allows the original sinusoid to pass almost unaffected.
• Many filters introduce a phase shift (horizontal displacement) to the output sinusoid, meaning it's delayed or advanced compared to the original.

Fourier Transform (FT) as a Filter Response Tool

• FT analyzes a filter's effect on each frequency.
• Its output (H(ω)) is a complex number representing the magnitude change (A) and phase shift (φ) caused by the filter at each frequency (ω).
• Magnitude (A) indicates how much a frequency is amplified/attenuated by the filter (F).
• Phase Shift (φ) indicates any delay or advancement in the timing caused by the filter (F).

Discrete Fourier Transform (DFT)

• Specifically for digital signals, or sampled data.
• DFT takes O(N²) operations.
  • H(k) = DFT at frequency k.
  • = Original Signal at time.
  • = Total number of samples.
  • = sinusoidal basis functions.
• Fast Fourier Transform (FFT)
  • An efficient algorithm to compute the DFT quickly.
  • FFT takes O(N log₂ N) operations.
• FT is applied to continuous signals, while DFT is used for discrete sampled signals.

Discrete Fourier Transform Filters

• Box-3 and Box-5 are smoothing filters (low-pass), blur the image by averaging neighboring pixel values.
• Linear Filter: smoothing filter that is similar to Box-3 but with slightly weights that emphasize center pixel more.
• Binomial Filter is a smoothing, and similar to the Gaussian filter and it is used for blurring while reduing noise.
• Sobel: an edge detection filter that emphasizes horizontal or vertical gradients in the image.
• Corner: detects corners in images, highlighting areas where intensity changes in multiple directions.

Two-Dimensional Fourier Transforms

• An image can be visualized as a 2D function of position (x and y coordinates).
• The 2D FT extends the concept to analyze its frequency content across both horizontal and vertical directions.
• Here, N and M are the width and the height of the image.

Applications of Fourier Transform

• Understanding Image Content: Provides insights into its characteristics.
  • High frequencies correspond to rapid changes like sharp details and edges.
  • Low frequencies represent slow, smooth variations and overall background.
• Image Enhancement:
  • Sharpening: Amplify high-frequency components by enhancing edges and other details in the image's Fourier Transform.
  • Blur Removal: The effect can undo blurring in the Fourier if the blur type is known.
  • Noise Removal (Denoising): Noise has high frequencies, reduced by removing high frequencies while keeping the edges and textures.

Evaluation of Image Denoising Algorithms

• PSNR (Peak Signal-to-Noise Ratio): A common algorithm to compare denoised image to the original images.
• SSIM (Structural Similarity Index): an algorithm to compare denoised image to original images that reflects the human perception.
• FLIP (Flicker Perception): Evaluates the smoothness of a video or image sequence by focusing on flicker or temporal artifacts.
• Using Neural Networks.
• No-reference assessment: Effectiveness measured by image denoising by original image is unknown.

Image Resizing and Applications

• Image resizing adjusts the dimensions of an image, with different reasons for doing so.
• Reasons:
  • Match output device resolution (printer, screen)
  • Reduce file size for storage/transmission
  • Optimize algorithm speed
  • Find objects at different scales (e.g., face detection)
  • Advanced image editing like seamless blending.

Image Resizing Techniques

• Upsampling: Using interpolation for enlarging images.
• Downsampling: Using decimation for shrinking images.
• Multi-Resolution Pyramids: For a structured set of resized images.

Interpolation for Upsampling

• To enlarge an image, an interpolation kernel convolved with the image that's filling in the blanks in a grid of pixels.
• Types of Kernels:
  • Linear: Simple but can create jagged edges.
  • Bicubic: Common choice, smoother results.
  • Windowed Sinc: Highest quality, but can introduce ringing.

Decimation or Downsampling Images

• To reduce image resolution by decimation.
• Decimation Process:
  • Convolution with a low-pass filter avoids aliasing caused by using high-frequency details.
  • Convolution is evaluated at every sample to optimize computation.

Common Filters for Decimation

• Common downsampling filters ( r=2):
  • Linear, Binomial, Cubic: From simple to complex filters.
• Tradeoffs with the use of algorithms:
  • Binomial leaves some aliasing, but is better compared to the use of Linear.
  • Cubic (a=-1) helps suppresses aliasing while causing ringing.
• Advanced Filters
  • QMF-9, JPEG2000 filters for specific tasks

Multi-Resolution Representations

• Multi-Resolution Analysis
  • The understanding of signals and images at different scales.
  • Varying Scales: analysis of large-scale features (overall shapes) with fine-scale details (edges and textures).
• Applications:
  • Image Compression: Store efficiently images with important details on across the scales.
  • Feature Detection: Detects key points/regions which are also key points that are informative in the images sizes and how they can be resized.

Image Pyramids Overview

• Structure: A hierarchical series of images, where each level is a lower-resolution version of one before it.
• Progressive Resolution Reduction
  • common, creating with ¼ pixels a level to another.
  • Filtering: Crucial to avoid aliasing artifacts when downsampling.
• Types: -Gaussian Pyramids: Reapeating process of smoothing and downsampling. -Laplacian Pyramids: Stores detail differences using layers from smoothing, allows reconstruction. -Wavelet Pyramids: Captures directional and detail of image, using different resolutions.

Decimation, Interpolation and Pyramids

• Downsampling using a pyramid and upsampling during reconstruction.
• Crucial the processes of using the filters for both are of quality!

Gaussian Pyramid

• Construction:
  • Repeated blurring of the image with a Gaussian filter.
  • Downsample to create subsequent levels by a amount of 2.
• Binomial Filter:
  • Good for balancing an approximation through efficiency and simplicity through good Gaussian blur.
• Applications:
  • Foundation for feature detection and other feature pyramids

Laplacian Pyramid

• Storage:
  • Storing each layer using the Gaussian levels
• Construction:
  • Create a lower res Gaussian level
  • Subtracting it from the higher resolution to get the Laplacian image.
• Perfect Reconstruction:
  • Laplacian with the smallest Gaussian will reconstruct the original image.

Frequency Response

• Filters
  • Image frequency is affected using certain filter to analyze.
  • Sharp Cutoff vs Aliasing:
• Ideal filters can be cut off and are harder. Simplers can be cut shorter and harder.
• Applications
• Dictate the choices while the artifacts sensitivity in the image based on computations.

Image Pyramids

• Applications
  • Coarse to Fire: Locate objects through refining the process
  • Multi resolution: Use of various blending that can produce high resolutions
  • Fractional scaling without blockiness (MIP)
  • Medical Whole Slide Imaging