Deep Neural Networks II - CNNs and RNNs

Deep Neural Networks II - CNNs and RNNs

• Kyung-Ah Sohn, Ajou University
• Deep neural networks, convolutional neural networks (CNNs), and recurrent neural networks (RNNs) are covered.

Table of Contents

• Convolutional Neural Networks (CNNs)
  • CNN architecture, training and regularization
  • Named CNNs
  • Transfer Learning
• Recurrent Neural Networks (RNNs)
  • Sequence-based prediction
  • Gated RNNs
  • Sequence-to-sequence problem

Convolutional Neural Networks (CNNs)

• CNNs are specialized neural networks for grid-like data such as images.
• They scale up neural networks for processing very large images and/or video sequences.
• Convolutions are used in CNNs.

Recurrent Neural Networks (RNNs)

• RNNs are useful for processing sequences of vectors.
• They use a recurrence formula at each time step for processing a sequence.
• The same function and set of parameters are used at every time step.
• RNNs can return a sequence as output or the last output.
  • They have various applications in natural language processing (NLP).
• RNNs suffer from the vanishing gradient problem, especially when sequences are long.

Applications of RNNs

• Sentiment classification
• Speech recognition
• Machine translation
• Text generation

Challenges of RNNs

• Defining network architecture to handle variable input lengths.
• Handling past information and using it for future prediction.

Example: Sequence Classification

• Input sequence is used to generate output.
• The output can be a classification or regression prediction based on various variables.

Sequence Classification: Input Encoding

• The input sequence (vector representations of input sequences) is encoded into hidden state.
• The input sequence is converted into a vector and used in calculation of the output.
• Common weights are used across all time steps.

Sequence Classification

• The entire sequence is encoded as the last hidden state.
• A classifier or regressor maps the encoding (the last hidden state or latent representation) to the output.

Recurrent Neural Network

• Recurrence formula is applied at each time step during the process of a sequence of vectors.
• This formula involves a new state, old state, and input vector at the time step.

RNN Output

• The recurrent layer can return a sequence as output.
• Another option for an output is the last output value.

Different Categories of Sequence Modeling

• One-to-one (e.g., image captioning)
• One-to-many (e.g., sentiment analysis)
• Many-to-one (e.g., machine translation)
• Many-to-many (e.g., video classification)

RNN is Hard to Train

• Real-world experiments, like those on language modeling, show that RNNs can sometimes be hard to train.

Exploding/Vanishing Gradient Problem in RNNs

• During backpropagation, the gradient can either explode or vanish, depending on the weights.
• The value of the tanh activation function is usually less than one.

Practical Measures to address RNN Training Issues

• Exploding gradients can be clipped to a threshold.
• Training can use truncated backpropagation through time.
• Learning rate can be adjusted.
• Vanishing gradients are harder to detect and resolve.
• Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs) are used to tackle this problem.

Long Short-Term Memory (LSTM)

• LSTMs overcome some of the short-term memory problems of standard RNNs
• LSTMs have a memory cell that functions as the hidden state of standard RNNs.
• A gating mechanism controls information flow.

Gating Mechanism

• A vector controls how much information will be kept or discarded.
• The sigmoid function's output value helps in this selection (between 0 and 1).

LSTM: Using Gates & Cell State

• Gradient vanishing is avoided by a new set of hidden states—cell state (C)—with a "highway" detouring the FC layer.
• Three types of gates (forget, input, output) control information flow.

Gated Recurrent Unit (GRU)

• GRUs are a more simplified architecture than LSTMs.
• GRUs combine the forget and input gates into a single update gate.
• GRUs merge the cell state and hidden state.
• GRUs usually have fewer parameters than LSTMs.

LSTM vs. GRU

• LSTMs and GRUs are commonly used gated RNN variants.
• LSTMs are a great default choice when speed and fewer parameters aren't primary considerations.

Common Variations of RNNs

• Bi-directional RNNs
• Deep (multi-layer) RNNs
• Handling vanishing gradients by introducing skip connections.

Example: LSTM for Sequence Classification

• Implementation in Keras provides information on how to use an embedding layer, LSTM layer, and a dense layer for sequences in a classification task.

Sequence-to-Sequence Problems

• Seq2seq problems, like machine translation, have different input/output sequence lengths.
• There isn't always a one-to-one correspondence between input and output tokens.
• An encoder-decoder structure can be used to address the differing input and output sequence lengths and non- one-to-one correspondence.

Encoder-Decoder Structure

• Encoder compresses the entire input sequence into a vector representation (embedding).
• Decoder generates the output from this embedding.
• This allows for variable length input/outputs.

Decoder RNN: Autoregressive Generation

• Autoregressive generation in decoders is done by using Softmax activation to generate the probabilities of each output token.
• Probability of the next token depends on previously generated tokens.

Information Loss in RNNs

• The entire sequence is encoded in a single embedding, causing information related to earlier inputs to be lost.
• Techniques to handle this information loss include attention mechanisms.

Attention Mechanism

• The attention mechanism allows the decoder to focus on important parts of the input sequence, and the encoder's hidden states.
• The context vector varies for each step of the decoder.

Attention Heatmap

• The attention heatmap shows the relative importance (weight) given to each input word when generating a target word in machine translation.

RNN Encoder-Decoder (with/without attention)

• Shows how the encoder compresses the entire input sequence into a fixed-length vector.
• Shows how the decoder uses the encoded vector's information during the generation of the output.
  • The use of attention helps the decoder focus on the relevant parts of the input sequence to produce the output sequence.

Attention Function

• Used for computations within a seq2seq RNN model.
• Q (query), K (key), and V (value) are inputs.
• Used to focus on the weights of the encoded-sequence parts when computing the encoded vector.
• The values of Q, K, and V have the same dimensionality.

Attention Methods

• Options for scoring similarities include dot-product attention, learnable weighted dot-product attention, or concatenation.

Attention-Based Seq2Seq Model

• Models dependencies without regard to their distance in the input/output sequence.
• Dealing with long sequences is a challenge and often hard to parallelize.

Real-World Success of RNNs and Transformers

• LSTMs and GRUs improved performance in machine translation tasks.
• Transformers have shown greater strength and wider adoption in real-world applications.

RNN: Summary

• RNNs are good for processing sequence data.
• Short-term memory problems can be mitigated with gating mechanisms.
• Multi-layer RNNs can be powerful but may need skip/dense connections
• Attention mechanisms can be vital for complex seq2seq problems.