Visual Object Recognition

• Visual Object Recognition is about learning visual categories and identifying new instances of those categories

Overview

• Recognition is the core problem of learning visual categories
• Any vision task fundamentally relies on the ability to recognize objects, scenes, and categories
• There as two types as seen by vision researchers; the specific case and the generic category case
• The specific case identifies a particular object, place, or person
• Examples of specific cases are: Carl Gauss's face, the Eiffel Tower, or a certain magazine cover
• At the category level, recognition is the recognition of different instances of a generic category as belonging to the same conceptual class
• Examples of category level recognition are: buildings, coffee mugs, or cars
• What sorts of categories can be recognized on a visual basis
• According to Rosch et al. (1976) and Lakoff (1987), basic level is:
• The highest level at which category members have similar perceived shape
• The highest level at which a single mental image can reflect the entire category
• The highest level at which a person uses similar motor actions for interacting with category members
• The level at which human subjects are usually fastest at identifying category members
• Basic-level categories are a good starting point for visual classification because they require the simplest visual category representations
• Category concepts below this basic level carry some element of specialization down to an individual level of specific objects, which require different representations for recognition
• Concepts above the basic level make some kind of abstraction and require additional world knowledge on top of the visual information
• The current standard pipeline for specific object recognition in computer vision relies on a matching and geometric verification paradigm
• For generic object categorization, it often includes a statistical model of appearance or shape learned from examples
• For the categorization problem, learning visual objects entails gathering training images of the given category, and then extracting or learning a model that can make new predictions for object presence or localization in novel images
• Models are often constructed via supervised classification methods, with some specialization to the visual representation when necessary
• The type of training data required as well as the target output can vary depending on the detail of recognition that is required
• The target task may be to name or categorize objects present in the image, to further detect them with coarse spatial localization, or to segment them by estimating a pixel-level map of the named foreground objects and the background

Challenges

• Matching and learning visual objects is challenging on a number of fronts
• Instances of the same object category can generate very different images, depending on confounding variables such as illumination conditions, object pose, camera viewpoint, partial occlusions, and unrelated background clutter
• Different instances of objects from the same category can also exhibit significant variations in appearance
• In many cases appearance alone is ambiguous when considered in isolation, making it necessary to model not just the object class itself, but also its relationship to the scene context and priors on usual occurrences.

Global Image Representations

• Writing down the intensity or color at each pixel in some defined order relative to a corner of the image is the most direct representation of an appearance pattern
• If the images are cropped to the object of interest and rather aligned in terms of pose, then the pixel reading at the same position in each image is likely to be similar for same-class examples
• Thus the list of intensities can be considered a point in a high-dimensional appearance space where the Euclidean distances between images reflect overall appearance similarity
• Most of the global representations lead to recognition approaches based on comparisons of entire images or entire image windows
• Such approaches are well-suited for learning global object structure
• Global Image Representations cannot cope well with partial occlusion, strong viewpoint changes, or with deformable objects

Local Feature Representations

• If having a model view of a (rigid) object, the task is to recognize whether this particular object is present in the test image and, if it is, where it is precisely located and how it is oriented
• Representing the image content by a collection of local features that can be extracted in a scale and rotation invariant manner addresses this task
• Those local features are first computed in both images independently
• The two feature sets are then matched in order to establish putative correspondences
• Due to the specificity of feature descriptors like SIFT (Lowe 2004) or SURF (Bay et al. 2006), the number of correspondences may already provide a strong indication whether the target object is likely to be contained in the image
• There will however be a number of mismatches or ambiguous local structures
• An additional geometric verification stage is applied in order to ensure that the candidate correspondences occur in a consistent geometric configuration
• The recognition procedure has 3 basic steps:
• Extract local features from both the training and test images independently
• Match the feature sets to find putative correspondences
• Verify if the matched features occur in a consistent geometric configuration
• Local invariant features purpose is to provide a representation that allows to efficiently match local structures between images
• Wanting to obtain a sparse set of local measurements that capture the essence of the underlying input images and that encode their interesting structure
• Feature extractors must fulfill two important criteria
• The feature extraction process should be repeatable and precise, so that the same features are extracted from two images showing the same object
• At the same time, the features should be distinctive, so that different image structures can be told apart from each other
• Its typically require a sufficient number of feature regions to cover the target object, so that it can still be recognized under partial occlusion
• The feature extraction pipeline:
• Find a set of distinctive keypoints
• Define a region around each keypoint in a scale- or affine-invariant manner
• Extract and normalize the region content
• Compute a descriptor from the normalized region
• Match the local descriptors

Keypoint Localization

• Finds a set of distinctive keypoints that can be reliably localized under varying imaging conditions, viewpoint changes, and in the presence of noise.
• The extraction procedure should yield the same feature locations if the input image is translated or rotated
• If considering a point lying in a uniform region, the exact motion can not be determined, since it cannot distinguish the point from its neighbors
• If considering a point on a straight line, its only have motion perpendicular to the line
• Keypoint detectors employ different criteria for finding such regions: the Hessian detector and the Harris detector

The Hessian detector

• Searches for image locations that exhibit strong derivatives in two orthogonal directions
• Based on the matrix of second derivatives, the so-called Hessian
• Derivative operations are sensitive to noise, we always use Gaussian derivatives which the derivative operation is combined with a Gaussian smoothing step with smoothing parameter σ
• The detector computes the second derivatives Ixx, lxy, and lyy for each image point, then searches for points where the determinant of the Hessian becomes maximal
• This search is usually performed by computing a result image containing the Hessian determinant values and then applying non-maximum suppression using a 3 × 3 window
• the search window is swept over the entire image, keeping only pixels whose value is larger than the values of all 8 immediate neighbors inside the window

The Harris detector

• Harris detector (Forstner & Gulch 1987, Harris & Stephens 1988) was explicitly designed for geometric stability
• It defines keypoints to be “points that have locally maximal self-matching precision under translational least-squares template matching" (Triggs 2004)
• These keypoints often correspond to corner-like structures
• The Harris detector proceeds by searching for points x where the second-moment matrix C around x has two large eigenvalues
• The matrix C can be computed from the first derivatives in a window around x, weighted by a Gaussian G(x, ˜σ)

Harris vs Hessian

• Harris locations are more specific to corners, while the Hessian detector also returns many responses on regions with strong texture variation
• Harris points are typically more precisely located as a result of using first derivatives rather than second derivatives and of taking into account a larger image neighborhood
• Harris points are preferable when looking for exact corners or when precise localization is required
• Hessian points can provide additional locations of interest that result in a denser coverage of the object