Visual Category Recognition

Overview

• Recognition is learning visual categories and identifying new instances
• Visual tasks rely on recognizing objects, scenes, and categories
• Two types of recognition exist: specific case and generic category case
• The specific case identifies instances of a particular object, place, or person like Carl Gauss's face or the Eiffel Tower
• The generic category case recognizes different instances of a conceptual class like buildings, coffee mugs, or cars
• An important question is what sorts of categories can be recognized visually

Basic Level Categories

• According to Rosch et al. (1976) and Lakoff (1987), the basic level is also at its:
  • Highest level, category members have similar perceived shape
  • Highest level, a single mental image reflects the entire category
  • Highest level, a person uses similar motor actions to interact with category members
  • Level at which human subjects are usually fastest at identifying category members
• Basic-level categories require the simplest visual category representations
• Category concepts below this basic level specialize down to individual objects and require different recognition representations
• Concepts above the basic level abstract and need additional world knowledge on top of visual data

Object Recognition Pipelines

• Computer vision uses a matching and geometric verification paradigm for specific object recognition
• Generic object categorization includes a statistical model of appearance/shape learned from examples
• Learning visual objects involves gathering training images, extracting or learning a model for predictions on object presence in new images
• Supervised classification methods construct these models and may require visual representation specialization
• Training data and target output type depend on required detail of recognition

Target Tasks

• Target tasks include naming or categorizing objects in an image
• Target tasks include detecting objects with coarse spatial localization
• Target tasks include segmenting objects by estimating a pixel-level map of named foreground objects and the background

Challenges in Visual Object Recognition

• Matching and learning visual objects include challenges
• Instances of the same category produce different images
• These result from confounding variables like illumination, object pose, camera viewpoint, partial occlusions, and unrelated background
• Objects from the same category vary in appearance
• Appearance alone is ambiguous, requiring models of the object class in relation to scene context and priors

Global Image Representations

• Represents appearance pattern by writing intensity or color at each pixel in defined order relative to a corner of the image
• Pixel readings, at the same position, in an image are often similar
• Intensities form a point in a high-dimensional appearance space
• Euclidean distances between images reflect overall appearance similarity
• Recognition uses comparing entire images or image windows
• Suited for learning global object structure, but struggle with strong viewpoint changes, partial occlusion, or deformable objects

Local Feature Representations

• Task is to recognize if a particular object, given a rigid model view, is present in test image, and its precise location and orientation
• This is solved by using image content as a collection of local features that are scale and rotation invariant
• Local features computed independently in both images
• Two feature sets are then matched to determine correspondences

Recognition Procedure

• Due to descriptor specificity like SIFT or SURF, the number of correspondences can help identify the target object
• Mismatches or ambiguous local structures are common
• An additional geometric verification stage ensures correspondences occur in correct geometric shape
• Basic steps of the recognition procedure:
• Extract local features from training and test images
• Match the feature sets to find correspondences
• Verify that the matched features occur in a geometric configuration that is consistent

Local Features and Description

• Local invariant features are a representation to efficiently match local structures between images
• This method obtains sparse set of local measurements that capture the essence of an image and encode structure
• Feature extractors need to meet the following criteria:
• The feature extraction process needs to be repeatable and precise which allows features to be easily extracted in other images
• The features need to be distinctive which allows the features to be told apart from each other
• Sufficient feature regions are required to cover the target object for recognition with partial occlusion

Feature Extraction Pipeline

• Find a set of distinctive keypoints
• Define a region around each keypoint in a scale- or affine-invariant manner
• Extract and normalize the region content
• Compute a descriptor from the normalized region
• Match the local descriptors

Keypoint Localization

• Find the set of distinctive keypoints that:
• Can be reliably localized under varying imaging conditions
• Can be reliably localized with viewpoint changes
• Can be reliably localized despite noise

Keypoint Detectors

• Feature extraction requires the same feature locations if the input is translated or rotated
• For a point in uniform region, it is not possible to determine its exact motion
• For a point on a straight line, it is only possible to measure motion perpendicular to the line
• Detectors focus on points that exhibit signals in two directions
• The Hessian and Harris detectors find such regions

The Hessian Detector

• The Hessian detector searches for image locations that exhibit strong derivatives in two orthogonal directions
• It is based on the Hessian matrix of second derivatives
• Derivative operations are sensitive to noise, so derivatives combine with a Gaussian smoothing step, using parameter σ
• The detector computes the second derivatives Ixx, Ixy, and Iyy for each image point
• Then detector searches for points where the determinant of the Hessian becomes maximal
• The detector computes a result image of Hessian determinant values and then applying non-maximum suppression using a 3 × 3 window
• Search window is swept over image, keeping pixels larger than all 8 immediate neighbors inside the window

The Harris Detector

• Harris detector (Forstner & Gulch 1987, Harris & Stephens 1988) designed for geometric stability
• Keypoints must have locally maximal self-matching precision under translational least-squares template matching (Triggs 2004)
• Keypoints can correspond to corner-like structures
• Harris detector searches for points x, where the second-moment matrix C around x has two large eigenvalues weighted by Gaussian G(x, ˜σ)

Harris and Hessian Detectors compared

• Harris locations are more specific to corners, while Hessian returns responses on regions with strong texture variation
• Harris points are more precisely located using first derivatives and a larger image neighborhood
• Harris points are preferable when exact corners or precise localization is required
• Hessian points provide additional locations of interest that result in a denser coverage of the object