Neural Network Architectures and Recognition Performance

Object Recognition and Neurons

• Studies on single neurons in the temporal lobe support the distributed code of object recognition
• Neurons are selective for complex objects, but this selectivity is often relative, not absolute

IT Neuron Selectivity

• IT neuron selectivity can appear arbitrary, responding to specific colors, textures, and shapes
• Cells with this selectivity likely provide inputs to higher-order neurons that respond to specific objects

Object Manifolds

• For neurons with small receptive fields, object manifolds are highly curved and "tangled" together
• This makes object recognition challenging, but can be achieved through categorization and identification

Object Recognition and Transformation

• Objects can be reliably categorized and identified even when transformed (spatially shifted or scaled)
• This is possible even when the classifier only saw each object at one particular scale and position during training

Neural Predictivity and Object Recognition

• Performance on object recognition tasks is correlated with neural predictivity
• Models that perform better on categorization tasks are also more likely to produce outputs closely aligned to IT neural responses

Hierarchical Linear-Nonlinear (HLN) Hypothesis

• The HLN hypothesis is consistent with various neural network architectures
• Specific parameter choices have a significant effect on a model's recognition performance and neural predictivity

Convolutional Neural Networks for Object Recognition

• In primates, the visual ventral pathway is critically involved in object recognition and extends from V1 to the IT cortex in the temporal lobe.
• The ventral visual pathway gradually "untangles" information about object identity.
• Classifier-based readout techniques can accurately read object identity from primate inferotemporal (IT) cortex with small populations of IT neurons (~300 units) over short time intervals (as small as 12.5 milliseconds).

Deep Convolutional Neural Networks (DCNNs)

• DCNNs are good candidates for models of the ventral visual pathway and have achieved near-human-level performance on challenging object categorization tasks.
• Core object recognition involves the ability to rapidly identify objects in the central visual field, in a single natural fixation (~200 ms), despite various image transformations (i.e., changes in viewpoint) and background.

Comparison with Primate Performance

• Monkey performance shows a pattern of object confusion that is highly correlated with human performance confusion pattern (0.78).
• Each behavioral metric computes a sensitivity (discriminability) index: d' = Z(HitRate) - Z(FalseAlarm-Rate), where Z is the standard z score.
• Object-level behavioral comparison reveals that human consistency is used to quantify the similarity between a model visual system and the human visual system with respect to a given behavioral metric (signatures).
• Image-level behavioral comparison shows that all leading DCNN models failed to replicate the image-level behavioral signatures of primates.

Limitations of DCNN Models

• Rhesus monkeys are more consistent with the archetypal human than any of the tested DCNN models (at the image level).
• Synthetic image-optimized models were no more similar to primates than ANN models optimized only on ImageNet, suggesting that the tested ANN architectures have one or more fundamental flaws that cannot be readily overcome by manipulating the training environment.
• DCNN models diverge from primates in their core object recognition behavior, suggesting that either the model architectural (e.g., convolutional, feedforward) and/or the optimization procedure (including the diet of visual images) that define this model subfamily are fundamentally limiting.

Recurrent Neural Networks

• Deep CNNs trained on object categorization are the best predictors of primate behavioral patterns across multiple core object recognition tasks.
• Unlike the primate ventral stream, these neural networks in this family are almost entirely feedforward and lack cortico-cortical, subcortical, and intra-areal recurrent circuits.
• The short duration (~200 ms) needed to accomplish accurate object identity inferences in the ventral stream suggests the possibility that recurrent circuit-driven computations are not critical for these inferences.

Time-Evolving IT Population Response

• To determine the time at which object identities are formed in the IT cortex, neural decode accuracy (NDAs) was estimated for each image, every 10 ms (from stimulus onset), by training and testing linear classifiers per object independently at each time bin.
• The term object solution time (or OST) refers to the time at which the NDA measured for each image reached the level of the behavioral accuracy of each subject (pooled monkey).
• The IT decode solutions for challenge images emerge slightly later than the solutions for the control images (average difference ~30 ms).
• The challenge image required an additional time of ~30 ms to achieve full solution compared with the control images, regardless of whether the animal was actively performing the task or passively viewing the images.