ELE888 - Final (Theory)

Convolutional Neural Networks (ConvNets or CNNs)

• CNNs are neural networks effective for image recognition and classification.
• CNNs have been successful in identifying faces, objects, and traffic signs, and in powering vision in robots and self-driving cars.
• CNNs contain four main operations: Convolution, Non-linearity (ReLU), Pooling/Sub Sampling, and Classification (fully connected layer).
• A channel is a component of an image; a standard digital camera image has three channels (red, green, blue), while a grayscale image has one channel.
• Convolution in CNNs primarily extracts features from the input image.
• Convolution preserves the spatial relationship between pixels using small squares of input data.
• Filters act as feature detectors, with the image and filters represented as numeric matrices.
• CNNs learn filter values during training, requiring specification of parameters like number of filters, filter size, and network architecture.
• The output matrix of a convolution operation is called the Convolved Feature or Feature Map.
• Feature Map size depends on depth (number of filters), stride (pixel shift over the input matrix; larger stride yields smaller feature maps), and zero padding.
• Zero padding, also known as wide convolution, controls feature map size, whereas narrow convolution lacks zero padding.

Non-Linearity

• Non-Linearity's output is the max of zero or the input.
• The purpose of Non-Linearity is to introduce non-linearity to the CNN to prepare it for real-world problems.
• Non-Linearity replaces all negative values in the feature map with zero.

Pooling Step

• Spatial Pooling (subsampling/downsampling) reduces dimensionality while retaining key information and can be Max, Average, or Sum.
• Pooling reduces the spatial size of the input representation.
• Pooling makes input representations smaller, more manageable, reduces parameters/computations to control overfitting.
• Pooling makes the network invariant to small transformations, distortions, and translations in the input image.
• Pooling helps achieve a scale-invariant representation ("equivariant").

Fully Connected Layer

• The Fully Connected layer is a Multi Layer Perceptron using a softmax activation function.
• "Fully Connected" means every neuron in the previous layer connects to every neuron on the next layer.
• Convolutional and pooling layers output high-level features. The Fully Connected layer classifies the input image into classes using these features.
• Adding a fully-connected layer is a cheap way of learning non-linear feature combinations, and the Convolution + Pooling layers act as Feature Extractors while the Fully Connected layer classifies.
• More convolution steps allow the network to recognize more complicated features.

Backpropagation

• Backpropagation is a technique for training neural networks which updates weights using gradient descent to minimize the error (difference between prediction and actual output).
• Gradient descent is an iterative optimization algorithm for finding the minimum of a function.
• Neural network training finds weights that minimize prediction error.
• Backpropagation uses gradient descent to adjust weights based on gradients of the loss function; the learning rate controls the weight adjustment amount in each iteration.
• A high learning rate may cause instability, while a low rate may slow down training.
• Backpropagation uses the chain rule to calculate gradients of the loss, enabling efficient calculation of how each weight change affects the network.
• Non-linear functions like ReLU, Sigmoid, or Tanh introduce non-linearity, enabling the network to learn complex patterns, with proper initialization avoiding vanishing or exploding gradients.
• Advantages include efficiency, optimization, flexibility, and adaptability, while disadvantages include high computational cost, susceptibility to overfitting and vanishing/exploding gradients, and the process getting stuck in local minima.

Naive Bayes

• Naive Bayes uses Bayes Theorem, which applies conditional probability to classify data.
• Types include Gaussian (normal distribution, continuous data), Multinomial (discrete data), and Bernoulli (binary/boolean features).
• Naive Bayes suits text classification tasks like spam detection due to its ability to handle high-dimensional, sparse datasets.
• The key assumption is that features are independent given the class label, which is often unrealistic but the model performs surprisingly well regardless.
• Bayes theorem is the backbone of the algorithm, used to compute the probability.
• The algorithm calculates priors (class probability) and likelihoods (probability of observing data given the class).
• It is simple, interpretable, and often used as a baseline model before more complex classifiers.
• It performs well with small datasets but struggles with highly correlated features and complex decision boundaries.
• The main strengths are its simplicity, speed, and effectiveness with high-dimensional or small datasets.
• Advantages include simplicity, speed, effectiveness with small datasets and many features, handling of continuous/categorical data, and low computational cost.
• Disadvantages include poor performance with correlated features, the need for many training samples, and the inability to model interactions between features directly.

Principal Component Analysis (PCA)

• PCA is a dimensionality reduction technique to reduce dataset features while retaining variance.
• It transforms original features into principal components, which are linear combinations of the original features.
• Principal components are the directions of maximum variance, with the first few capturing most of the data's variance.
• PCA is a linear transformation and may not perform well on data with non-linear relationships.
• PCA assumes variance indicates information and is effective when data contains correlated features.
• Advantages include reduced dimensionality, improved model performance, removal of correlated features, enhanced visualization, and data compression.
• Disadvantages include the inability to capture complex non-linear relationships, sensitivity to outliers, and high computational cost for large datasets.
• It reduces computational complexity and noise by transforming data into orthogonal components.
• It works best for linearly separable data and is most effective when features are correlated but is sensitive to outliers and less effective for significant non-linearities.
• A Covariance matrix describes all relationships between pairs of measurements.
• A larger covariance indicates a larger correlation, while 0 covariance signals entirely un-correlated data.

Applying Machine Learning Algorithms

• When a learning algorithm yields large prediction errors, try getting more training samples or adding additional features.
• A diagnostic test can provide insight into the algorithm's issues and how to improve performance, while being time-consuming, can benefit performance in the long run.
• If a learning algorithm suffers from high bias, getting more training data will not help much.
• If a learning algorithm suffers from high variance, getting more training data is likely to help.
• If a learning algorithm makes large errors, try getting more training samples (fixes high variance), smaller sets of features (fixes high variance), or additional features (fixes high bias).
• Small neural networks with fewer parameters are more prone to underfitting but are computationally cheaper.
• Large, more complex neural networks with more parameters are more prone to overfitting and are computationally more expensive.