COMP9444 Week 9a: Autoencoders

Study Notes

COMP9444: Neural Networks and Deep Learning - Week 9a: Autoencoders

Autoencoder Networks: Networks trained to reproduce their input, often using a compressed representation (bottleneck). Activations typically pass through a bottleneck.
Regularized Autoencoders: Variations of autoencoders that add extra loss terms to encourage latent variables to follow a specific distribution or meet other objectives. Dropout, sparsity, contractivity, denoising, and variational autoencoders are examples.
Stochastic Encoders & Decoders: Decoders define conditional probability distributions for outputs given latent variables. Encoders can also be viewed as conditional probability distributions of latent variables given inputs.
Generative Models: Autoencoders can be used to generate new data points similar to existing data. Variational Autoencoders use explicit models to achieve this. Other models use implicit processes (e.g., Generative Adversarial Networks, GANs).
Variational Autoencoders (VAEs): These aim to maximize the log probability of the target variables, encouraging latent variables (z) drawn from conditional distributions to create outputs that closely match the originals.
Convolutional Networks for Generating Images: Networks trained to generate images typically consist of convolutional layers for downsampling followed by upsampling in the decoding process.
Autoencoders as Pretraining: Autoencoders can be used for initializing weights in other networks. Remove the decoder portion and replace with a classification layer, then train via backpropagation
Sparse Autoencoders: Regularize autoencoders via a penalty term, by penalizing the sum of absolute values of activations in hidden layers (L₁ - regularization).
Contractive Autoencoders: The L²-norm is used on the derivatives of hidden units with respect to inputs in a regularization term. This penalizes significant changes in hidden unit activations when input data is slightly varied.
Denoising Autoencoders: Add noise to inputs, trained on recovering the original input, encouraging the network to learn more robust features.
Loss Functions and Probability: Different loss functions (e.g. squared error, cross-entropy, softmax) represent different probability distributions (e.g., Gaussian, Bernoulli, Boltzmann).

Hopfield Network and Boltzmann Machine

Hopfield Network & Boltzmann Machine: These are networks used for image storage and retrieval. They are based on energy functions. Images are stored as weights in an energy function.
Image Representation: Pixels are represented as +1 or -1, white (+1) or black (-1).
Deterministic Update: In Hopfield networks, pixels are updated based directly on their neighbours (e.g. energy function).
Stochastic Update: In Boltzmann machines, pixels are updated probabilistically using a sigmoid activation function based on their neighbours and bias values. New similar images can be generated via stochastic updates

Recall: Encoder Networks

Encoder Networks Function: These networks create compressed representations of inputs via a "bottleneck". This process is analogous to the N-M-N task.
Hidden Unit Representations: Investigating how representations are organized and learned in hidden layers of these networks.

Autoencoder Networks

Bottleneck Layer: The core of autoencoders is a compressed representation to extract core features from input data. Forces data to be simplified into salient features.
Input & Output Layers: These layers allow the autoencoder to reconstruct the input data as closely as possible.
Fully Connected Layers: Used as foundational parts of the network.

Variational Autoencoders (20.10.3)

Gaussian Distribution for z: The encoder produces a mean and standard deviation for a Gaussian distribution for latent variable z, providing a probability distribution allowing for flexibility.
Maximizing Log Probability: The system is trained to maximize the log probability of the input data given the latent variables (z) generated by the encoder and reconstructed by the decoder.
KL divergence: A concept to measure how different two probability distributions are from each other. Used in VAE to encourage latent variables to closely approximate the prior distribution (e.g. normal).

Regularized Autoencoders (further subtypes)

Autoencoders with Dropout: Dropout regularizes a network by randomly deactivating nodes during training to prevent them from becoming overly reliant on each other.

Entry and KL Divergence

Entropy: The average amount of information contained in a probability distribution.
KL Divergence: A measure of the difference between two probability distributions. It quantifies the extra bits needed for transmission of samples if the wrong distribution is used.
Gaussian Distribution: A standard normal distribution is a common prior for the latent variable, often encouraging the latent variable in a VAE to be effectively centred at zero and spreading evenly across each dimension (represented by covariance matrices in each latent variable).