30 Questions
What is a major issue with adding more fully connected (FC) layers to a neural network?
Network becomes arbitrarily complex
To counter the problem of memorizing data when adding more layers to a network, what do we want instead?
Layers with structure
Which aspect makes it challenging to manage weights and matrix multiplications when adding more FC layers?
Using the same weights for different parts of the image
What feature of Convolutional Neural Networks (CNNs) distinguishes them from fully connected neural networks?
Structure in layers
What is one consequence of going further deeper in network complexity without proper structure in the layers?
Optimization becomes hard
Why do Convolutional Neural Networks (CNNs) use weight sharing?
To ensure the same weights are used for different image parts
What is the main purpose of the LeNet-5 architecture described in the text?
Recognition of hand-written digits
In the LeNet-5 architecture, what layer typically follows the Convolution-ReLU-Pool sequence?
Flatten
What kind of propagation is involved in training a CNN with Keras as mentioned in the text?
Forward and Backward Propagation with Gradient Descent
What is the purpose of the 'Linear(120→80)' layer in the LeNet-5 architecture?
Reducing the number of parameters
Which layer in the LeNet-5 architecture would be responsible for converting a 5x5x16 output to 120 units?
Flatten
When was the LeNet-5 architecture created according to the text?
1998
What was the purpose of LeNet-5 architecture created by Yann LeCun in 1998?
Handwritten digits recognition
What is the function of the ReLU layer in the LeNet-5 architecture?
Apply a non-linear activation function
In the LeNet-5 architecture, what does the 'Convolution-2 (f=5, k=6) 62828' layer indicate?
6 filters with 5x5 kernel applied to a 28x28 feature map
What is the purpose of the 'MaxPool-2 (f=2, s=2) 61414' layer in the LeNet-5 architecture?
Downsample the feature maps
What happens in the 'Linear(12080) 80' layer of the LeNet-5 architecture?
Reduction of 120-dimensional feature vectors to 80 dimensions
Which layer comes immediately after the 'MaxPool (f=2, s=2) 61414' in the LeNet-5 architecture?
Flatten 5516
Who authored the groundbreaking research paper 'Cognitron: A self-organizing multilayered neural network'?
Kunihiko Fukushima
Which research paper introduced the 'Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position'?
Kunihiko Fukushima
Who were the authors of the paper titled 'Learning representations by backpropagating errors'?
D.E.Rumelhart, G.E.Hinton, and R.J William
Who were the authors of the research paper that applied Gradient-based learning to document recognition?
Yann LeCun, Leon Bottou, Yoshua Bengio, Thomas Haffner
Which publication introduced 'ImageNet Classification with Deep Convolutional Neural Networks'?
Krizhevsky, Sutskever, Hinton
Which classic architecture is associated with the representation [Conv, ReLU, Pool]*N, flatten, [FC, ReLU]*N, FC, Softmax?
LeNet-5
What is the purpose of padding in image processing?
Add extra pixels at the image boundary
In convolution operations, what does 'Stride' refer to?
The amount by which the filter shifts over the input
What determines the dimensions of the output feature map in convolution operations?
Amount of zero padding
How does the input volume affect the output volume in convolutional layers?
Output volume is calculated based on the input volume
What is the formula for calculating the output width in convolutional operations?
$\text{Output width} = (\text{Input width} - \text{Filter size} + 2 \times \text{Padding}) / \text{Stride} + 1$
What aspect of convolutional layers is determined by the hyper-parameter 'K'?
Number of filters
Study Notes
Research Breakthroughs in CNN
- Fukushima, K. (1975) introduced the concept of "Cognitron: A self-organizing multilayered neural network" in Biological Cybernetics, 20(3-4), 121-136.
- Fukushima, K. (1980) published "Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position" in Biological Cybernetics.
- D.E.Rumelhart, G.E.Hinton, and R.J William (1986) introduced "Learning representations by backpropagating errors" in Nature, Vol 323.
Important Publications for Practical Implementation
- LeCun, Bottou, Bengio, Haffner (1998) applied gradient-based learning to document recognition, introducing LeNet-5.
- Krizhevsky, Sutskever, Hinton (2012) introduced ImageNet Classification with Deep Convolutional Neural Networks, also known as AlexNet.
Case Studies - LeNet-5 Architecture
- Classic Architecture-LeNet-5: [Conv, ReLU, Pool]*N,→ flatten,[FC, ReLU]*N, FC → Softmax
- Layer Output Size and No. of Parameters:
- Convolution-2 (f=5, k=6): 62828, 665*5
- ReLU: 62828
- MaxPool-2 (f=2, s=2): 61414
- Flatten: 5516 (120), 665*5
- Linear(120→80): 80, 120*80
Challenges for Feature Extraction
- Illumination
- Deformation
- Occlusion
- Background Clutter
- Intra-class Variation
Why Not Add More FC Layers?
- Adding more layers makes the network arbitrarily complex
- Going further deep may have issues:
- No structure
- Forcing network to memorize data instead of learning
- Optimization becomes hard (managing weights and matrix multiplications)
- Performance drops as we move further deep → Underfitting
Convolutional Neural Networks (CNNs) Introduction
- No Brain stuff
- CNNs are a special case of fully connected neural network
Convolution Operations
- Demo with stride = 1 and no padding
- Filter dimensions: 3 × 3
- Output feature map dimensions: 3 × 3
- For images with multi-channels (e.g., color images)
- Hyper-parameters in ConvLayer: Number of filters (K), Size of filters (F), the Stride (s), and amount of zero padding (P)
- Formula for output dimensions: 𝑾𝒐𝒄 = 𝑯𝒐𝒄 = 𝑾𝒊𝒏 −𝑭+𝟐𝑷 𝑺 𝑯𝒊𝒏 −𝑭+𝟐𝑷 𝑺 +𝟏 +𝟏
Test your knowledge on image challenges related to CNN, including illumination, deformation, occlusion, background clutter, and intra-class variation. Explore key concepts and factors affecting feature extraction in image data.
Make Your Own Quizzes and Flashcards
Convert your notes into interactive study material.
