GRU vs LSTM: Key Differences Explained with Quiz & Flashcards

What problem did Hochreiter and Schmidhuber attempt to solve when they developed LSTM?

What is the main difference between a simple RNN and an LSTM?

What is the function of the multiplicative input gate unit in an LSTM?

What is the purpose of the constant error carrousel in LSTM?

What is stored in the long-term state memory C(t-1)?

What determines which part of the long memory is removed?

What is the advantage of using LSTM over simple RNN?

What is the purpose of the multiplicative output gate unit?

When was the LSTM architecture developed?

What is the name of the architecture that allows for constant error flow?

RNN with GRU and LSTM

In RNN, h(t-1) is the short-term state memory, where we can either forget or pass to the next step using the output gate.

GRU (Gated Recurrent Unit)

The sigmoid layer outputs numbers between 0 and 1, describing how much of each component should be let through.
The output gate determines how much of the past information (from previous time steps) needs to be passed along to the future.
The update gate (Γu) helps the model determine how much of the past information needs to be passed along to the future.
The reset gate (Γr) decides how much of the past information to forget.
Γu and Γr are calculated using sigmoid functions with weights and biases.
Ct is the new candidate values that could be added to the state.
Ct is calculated using the update gate, reset gate, and the previous state.

LSTM (Long Short-Term Memory)

LSTM was developed by Hochreiter and Schmidhuber in 1997 to address the vanishing gradient problem.
LSTM is a variant of SimpleRNN that adds a way to transport information over many time steps.
LSTM uses gates to protect the memory contents from perturbation by irrelevant inputs and outputs.
The input gate, output gate, and forget gate are used to regulate the flow of information in the LSTM cell.

This quiz covers the basics of Recurrent Neural Networks (RNN) with GRU and LSTM layers, including the input and output gates, short term and long term state memories. It's a perfect resource for students of Prof. Slimane LARABI at USTHB.