Image Segmentation and CNNs

• Computer Vision presentation by Naeemullah Khan, [email protected], February 19, 2025

Learning Outcomes

• Understand image segmentation fundamentals and its importance.
• Understand how Convolutional Neural Networks (CNNs) are adapted for segmentation tasks.
• Understand different upsampling techniques in segmentation models.
• Understand the role of residual connections and the U-Net architecture in segmentation.
• Differentiate between instance and panoptic segmentation.

Image Classification

• Image Classification is a core task in Computer Vision

Computer Vision Tasks

• Classification involves assigning a single label to an entire image, lacking spatial extent
• Semantic Segmentation classifies each pixel in an image into a predefined set of categories, resulting in a pixel-wise labeling of the image
• Object Detection identifies and localizes multiple objects within an image by drawing bounding boxes around each object
• Instance Segmentation is similar to object detection, but instead of providing bounding boxes, it delineates each object instance at the pixel level

Semantic Segmentation

• With paired training data, each pixel is labeled with a semantic category.
• At test time, each pixel of a new image gets classified.
• Classifying each pixel independently is impossible without considering context.

Semantic Segmentation Idea: Sliding Window

• A sliding window approach classifies the center pixel with a CNN, extracting patches around the pixel of interest
• The sliding window approach is very inefficient due to not reusing shared features between overlapping patches

Semantic Segmentation Idea: Convolution

• Instead encode the entire image with a conv net, and that does semantic segmentation on top
• Classification architectures reduce feature spatial sizes to go deeper, but semantic segmentation requires the output size to be the same as input size.

Semantic Segmentation Idea: Fully Convolutional

• Design a network with convolutional layers without downsampling operators to make predictions for pixels all at once
• Fully Convolutional networks convolutions at the original image resolution can be very expensive
• Design the network with downsampling and upsampling inside
  • Med-res: D₂ x H/4 x W/4
  • Low-res: D₃ x H/8 x W/8
  • High-res: D₁ x H/2 x W/2

In-Network Upsampling: Unpooling

• Nearest Neighbor unpooling duplicates the values resulting in a blocky output
• "Bed of Nails" unpooling places the input values in the top-left corner and fills the rest with zeros

In-Network Upsampling: Max Unpooling

• Max Unpooling uses the positions from pooling layer and remembers which element was max

Learnable Upsampling: Transposed Convolution

• Normal Convolution: a 3x3 convolution with stride 2 and pad 1.
• In normal convolution, the filter moves 2 pixels in the input for every one pixel in the output. -Strided convolution can be interpreted as "learnable downsampling".
• Transposed Convolution: a 3x3 transposed convolution, stride 2, pad 1
• Input gives weight for filter.
• Filter moves 2 pixels in the output for every one pixel in the input.
• Strided gives ratio between movement in output and input
• There are overlapping outputs, where there is a sum

Mathematical Definition: Transposed Convolution

• The output O(x, y) of a transposed convolution is computed as: O(x, y) = ∑ I(i,j) ⋅ K(x − i ⋅ s, y − j ⋅ s)
  • I(i,j) is the input value at (i,j)
  • K(x', y') is the kernel value at (x', y')
  • s is the stride

Transposed Convolution Output Size Formula

• Hout = (Hin - 1) × stride[0] – 2 × padding[0] + dilation[0] × (kernel_size[0] – 1) + output_padding[0] + 1
• Wout = (Win - 1) × stride[1] — 2 × padding[1] + dilation[1] × (kernel_size[1] — 1) + output_padding[1] + 1
• Hout, Wout are the Output height and width.
• Hin, Win are the Input height and width.
• Stride is the Step size of the filter movement.
• Padding is the Number of pixels added around the input.
• Dilation is the Spacing between kernel elements.
• Kernel size is the Size of the convolution filter.
• Output padding is the Additional padding applied to the output.

Semantic Segmentation Idea: Fully Convolutional

• Design network as a bunch of convolutional layers, with downsampling and upsampling inside the network!
  • Med-res: D₂ x H/4 x W/4
  • Low-res: D₃ x H/4 x W/4
  • High-res: D₁ x H/2 x W/2
• Downsampling: Pooling, strided convolution
• Upsampling: Unpooling or strided transposed convolution

Can We Do Better?

• Downsampling-then-upsampling approach works well for semantic segmentation.
• Important details and spatial information may be lost during downsampling.
• Introduce residual connections to preserve spatial information.

Residual Connections in Segmentation

• Connect features from downsampling layers to upsampling layers.
• Help recover lost spatial details and improve segmentation accuracy.
• Two Types of Residuals:
  • Addition: Adds features from the encoder to the decoder element-wise
  • Concatenation: Concatenates features from the encoder to the decoder along the channel dimension.
• Concatenation is often better because it retains more feature information from the encoder.
• Concatenation might be harder to implement because it requires aligning input and output shapes.

U-Net

• With residual connections, the architecture is called U-Net.
• The architecture resembles the shape of the letter "U"
• Features are downsampled in the encoder and upsampled in the decoder, with skip connections in between.
• U-Net is widely used for segmentation tasks, especially in biomedical imaging.

Using a Pretrained U-Net

• The encoder can use a pretrained backbone (e.g., ResNet, EfficientNet).
• Useful features learned on large datasets (e.g., ImageNet).
• Only the decoder is trained from scratch for segmentation-specific tasks.

Semantic Segmentation

• Label each pixel in the image with a category label
• Does not differentiate instances, only care about pixels

Instance Segmentation

• Separates object instances, but only things

Panoptic Segmentation

• Labels all pixels in the image (both things and stuff)