Deep CNNs vs Ventral Stream in Image Recognition

Deeper CNNs and Ventral Stream

• Deeper CNNs predict IT neural responses at late phases (150-250 ms) more accurately than 'regular-deep' models like AlexNet.
• This suggests that deeper CNNs might be approximating 'unrolled' versions of the recurrent circuits of the ventral stream.
• Deeper CNNs have a reduced number of challenge images, and the remaining challenge images show longer OSTs in the IT cortex.

CORnet Model

• CORnet is a four-layered recurrent neural network model with within-area recurrent connections and shared weights.
• The top layer of CORnet is comparable to IT and has higher IT predictivity for the late phase of IT responses.
• Pass 1 and pass 2 of CORnet are better predictors of early time bins (relevant for control images), while late passes (especially pass 4) are better at predicting late phases of IT responses (crucial for challenge images).

Recurrent Computations

• Recurrent computations act as additional nonlinear transformations of the initial feedforward during core object recognition.
• Deeper CNNs, such as Inception-v3, v4, and ResNet-50, are better models of the behaviorally critical late phase of IT responses due to the introduction of more nonlinear transformations.

Image Completion and RNN

• Recurrent computations enable pattern completion, which allows recognition of poorly visible or occluded objects.
• The visual system can make inferences even when only 10-20% of the object is visible.

Backward Masking

• Backward masking disrupts recognition of partially visible objects by interrupting any additional, presumably recurrent, processing of the image.

Feed-Forward Models and Occlusion

• Standard feed-forward models, such as AlexNet, are not robust to occlusion and their performance declines at limited visibility.

RNN Models

• Recurrent Neural Networks improve recognition of partially visible objects, with the RNNh model demonstrating a significant improvement over the standard AlexNet.
• Attractor networks, such as the Hopfield network, can perform pattern completion.
• The RNNh model's performance and correlation with humans saturate at around 10-20 time steps, consistent with the physiological responses to heavily occluded objects arising at around 200 ms.

Backward Masking and RNN Performance

• Presenting a mask reduces RNN performance, reproducing the effect of backward masking on human performance.

Unsupervised Neural Networks

• Unsupervised models are trained on ImageNet, a dataset of millions of category-labeled images, which is implausible for human infants and nonhuman primates
• Supervised DCNN cannot explain how representations are learned in the brain
• Unsupervised learning algorithms aim to learn representations from natural statistics without high-level labeling

Human Data vs. Standard Image Databases

• Human data is continuous and egocentric, whereas standard image databases are not
• Human input is multimodal, whereas model input is often unimodal
• Humans may rely on different inductive biases, allowing for more data-efficient learning
• Humans may enlarge their initial dataset by using already encountered instances to create new instances during offline states (i.e., imagination, dreaming)

Unsupervised Learning Algorithms

• Local Aggregation (LA) method: optimizes to push the current embedding vector closer to its close neighbors and further from its background neighbors
• Multi-dimensional scaling (MDS) algorithm: used to visualize the embedding space and shows classes with high and low validation accuracy

Contrastive Embedding Methods

• Yield high-performing neural networks
• Outperform other unsupervised methods and even category-supervised models in several tasks, including object position and size estimation
• Equaled or outperformed category-supervised models in several tasks

Comparison to Neural Data from Macaque Cortex

• Unsupervised neural network models were compared to neural data from macaque V1, V4, and IT cortex
• Unsupervised methods were significantly better than the untrained baseline at predicting neural responses in Area V1
• Only a subset of methods achieved parity with the supervised model in predictions of responses in Area V4
• Only the best-performing contrastive embedding methods achieved neural prediction parity with supervised models in Area IT

Deep Contrastive Learning on First-Person Video Data from Children

• ImageNet dataset diverges significantly from real biological data streams
• SAYCam dataset is a better proxy of the real infant data stream, containing head-mounted video camera data from three children
• Contrastive unsupervised learning is robust enough to handle real-world developmental video streams such as SAYCam
• VIE algorithm is an extension of LA to video and achieves state-of-the-art results on a variety of dynamic visual tasks

Partial Supervision

• Semisupervised learning leverages small numbers of labeled datapoints in the context of large amounts of unlabeled data
• Local label propagation (LLP) algorithm embeds datapoints into a compact embedding space and infers pseudolabels of unlabeled images from those of nearby labeled images
• LLP jointly optimizes to predict inferred pseudolabels while maintaining contrastive differentiation between embeddings with different pseudolabels
• Semisupervised models lead to representations that are substantially more behaviorally consistent than purely unsupervised methods, although a gap to supervised models remains