RNN with GRU and LSTM Layers

Questions and Answers

The gradient of fuite problem

The ability to learn long-term dependencies

To protect the memory contents from perturbation

To allow for constant error flow Signup and view all the answers

The information to be forgotten or passed Signup and view all the answers

The forget gate Signup and view all the answers

Ability to learn long-term dependencies Signup and view all the answers

To protect other units from perturbation Signup and view all the answers

1997 Signup and view all the answers

Constant Error Carrousel (CEC) Signup and view all the answers

Study Notes

In RNN, h(t-1) is the short-term state memory, where we can either forget or pass to the next step using the output gate.

The sigmoid layer outputs numbers between 0 and 1, describing how much of each component should be let through.
The output gate determines how much of the past information (from previous time steps) needs to be passed along to the future.
The update gate (Γu) helps the model determine how much of the past information needs to be passed along to the future.
The reset gate (Γr) decides how much of the past information to forget.
Γu and Γr are calculated using sigmoid functions with weights and biases.
Ct is the new candidate values that could be added to the state.
Ct is calculated using the update gate, reset gate, and the previous state.

LSTM was developed by Hochreiter and Schmidhuber in 1997 to address the vanishing gradient problem.
LSTM is a variant of SimpleRNN that adds a way to transport information over many time steps.
LSTM uses gates to protect the memory contents from perturbation by irrelevant inputs and outputs.
The input gate, output gate, and forget gate are used to regulate the flow of information in the LSTM cell.