RNN Processing Sequences Using RNNs and CNNs PDF

Chapter 15: Processing Sequences Using RNNs and CNNs Tsz-Chiu Au [email protected] Ulsan National Institute of Science and Technology (UNIST) South Korea  Recurrent Neural Networks Recurrent neural networks (R...

Chapter 15: Processing Sequences Using RNNs and CNNs Tsz-Chiu Au [email protected] Ulsan National Institute of Science and Technology (UNIST) South Korea  Recurrent Neural Networks Recurrent neural networks (RNNs) is a class of nets that can predict the future. » They can analyze time series data such as stock prices. In this chapter, we will study » The fundamental concepts underlying RNNs. » How to train them using backpropagation through time. » How to use them to forecast a time series. » How to cope with unstable gradients and a (very) limited short-term memory. » A CNN architecture called WaveNet that is capable of processing time series data as well as RNNs.  Recurrent Neurons and Layers A recurrent neural network looks very much like a feedforward neural network, except it also has connections pointing backward. A recurrent neuron unrolled through time: At each time step t (also called a frame), this recurrent neuron receives the inputs x(t) as well as its own output from the previous time step, y(t–1). » In the first time step, the input is 0  A Layer of Recurrent Neurons At each time step t, every neuron receives both the input vector x(t) and the output vector from the previous time step y(t–1). Each recurrent neuron has two sets of weights: Wx for the inputs x(t) and Wy for the outputs of the previous time step, y(t–1).  Memory Cells A recurrent neuron has memory because its output is a function of all the inputs from previous time steps. A part of a neural network that preserves some state across time steps is called a memory cell (or simply a cell). A single recurrent neuron, or a layer of recurrent neurons, is a very basic cell, capable of learning only short patterns. » To learn longer patterns, a more powerful type of cells is needed. A cell’s state at time step t, denoted h(t) (the “h” stands for “hidden”), is a function of some inputs at that time step and its state at the previous time step: h(t) = f(h(t–1), x(t)). The output at time step t, denoted y(t), is also a function of the previous state and the current inputs.  Input and Output Sequences Sequence-to-sequence network » E.g., predicting time series such as stock prices Sequence-to-vector network » E.g., feed the network a sequence of words corresponding to a movie review and output a sentiment score Vector-to-sequence network » E.g., the input could be an image, and the output could be a caption for that image. Encoder–Decoder » E.g., translating a sentence from one language to another. » Feed the network a sentence in one language, the encoder would convert this sentence into a single vector representation, and then the decoder would decode this vector into a sentence in another language.  Training RNNs Backpropagation through time (BPTT) » First, forward pass through the unrolled network » Second, the output sequence is evaluated using a cost function C(Y(0), Y(1),...Y(T)) § The cost function can ignore some outputs » Third, the gradients of that cost function are then propagated backward through the unrolled network » Fourth, the model parameters are updated using the gradients computed during BPTT. § Since the same parameters W and b are used at each time step, backpropagation will do the right thing and sum over all time steps.  Forecasting a Time Series A time series is a sequence of data, one per time step. » A univariate time series is a time series that have one value per time step. » A multivariate time series is a time series that have multiple values per time step. Forecasting is a task of predicting future values. » E.g., forecast the value at the next time step (represented by the X) in the following graphs Imputation is a task of predicting missing values from the past. Generating Time Series for Experiments In this chapter, instead of using real time series data in the real world, we will consider the time series generated by this function: The function returns a NumPy array of shape [batch size, time steps, 1], where each series is the sum of two sine waves of fixed amplitudes but random frequencies and phases, plus a bit of noise. » In general, the input features are represented as 3D arrays of shape [batch size, time steps, dimensionality], where dimensionality is 1 for univariate time series and more for multivariate time series. Let’s create a training set, a validation set, and a test set.  Baseline Metrics We like to compare our RNN models with some baseline methods to see whether the RNN models perform as well as we expect. Baseline 1: naive forecasting » Use the last value in a series to predict the next value. » It gives a mean squared error of about 0.020 in our previous example. Baseline 2: linear model (implemented as a fully connected network) » Use a simple Linear Regression model so that each prediction will be a linear combination of the values in the time series » If we compile this model using the MSE loss and the default Adam optimizer, then fit it on the training set for 20 epochs and evaluate it on the validation set, we get an MSE of about 0.004.  Implementing a Simple RNN Let’s build a very simple RNN and compare it with the baseline methods. It just contains a single layer with a single neuron. By default, the SimpleRNN layer uses the hyperbolic tangent activation function. The initial state h(init) is set to 0. The neuron computes a weighted sum of these values and applies the hyperbolic tangent activation function to the result, and this gives the first output, y0. In a simple RNN, this output is also the new state h0. This new state is passed to the same recurrent neuron along with the next input value, x(1). The process is repeated until it returns y49. By default, recurrent layers in Keras only return the final output. To make them return one output per time step, you must set return_sequences=True If you compile, fit, and evaluate this model (just like earlier, we train for 20 epochs using Adam), you will find that its MSE reaches only 0.014 » better than the naive approach but it does not beat the simple linear model. » Reason: this simple RNN has just three parameters whereas the simple linear model has 51 parameters.  Trend and Seasonality There are other models for forecasting time series » E.g., weighted moving average models and autoregressive integrated moving average (ARIMA) models. Some of them require you to first remove the trend and seasonality. » The known pattern of the time series should be ignored first. After the model is trained and makes predictions, you would have to add the trend and the seasonal pattern back to get the final predictions. When using RNNs, it is generally not necessary to do all this, but it may improve performance in some cases, since the model will not have to learn the trend or the seasonality.  Deep RNNs To implement a deep RNN with multiple layers of cells: Note that you must set return_sequences=True for all recurrent layers except the last one. If you compile, fit, and evaluate this model, you will find that it reaches an MSE of 0.003 (i.e., better than the linear model) The last layer is too simple, and we can replace it with a Dense layer  Forecasting Several Time Steps Ahead To predict not just the value at the next time step but also the next 10 values » One simple way is to use the trained model to predict the next value, then add that value to the inputs, and use the model again to predict the following value, and so on. In this method, the errors might accumulate over time. » We get an MSE of about 0.029, which is even worse than the naive forecasting (MSE of about 0.223) and the linear model (MSE of about 0.0188). Still, if you only want to forecast a few time steps ahead only on more complex tasks, this approach may work well. Forecasting Several Time Steps Ahead (cont.) The second option is to train an RNN to predict all 10 next values at once. » We can still use a sequence-to-vector model, but it will output 10 values instead of 1. » Now we just need the output layer to have 10 units instead of 1 The MSE for the next 10 time steps is about 0.008. » Much better than the linear model But we can still do better » instead of training the model to forecast the next 10 values only at the very last time step, we can train it to forecast the next 10 values at each and every time step. § i.e., turn this sequence-to-vector RNN into a sequence-to-sequence RNN. » The advantage of this technique is that the loss will contain a term for the output of the RNN at each and every time step, not just the output at the last time step. Forecasting Several Time Steps Ahead (cont.) At time step 0 the model will output a vector containing the forecasts for time steps 1 to 10, then at time step 1 the model will forecast time steps 2 to 11, and so on. To turn the model into a sequence-to-sequence model, » we must set return_sequences=True in all recurrent layers (even the last one) » we must apply the output Dense layer at every time step. Keras offers a TimeDistributed layer, which reshapes the inputs for the wrapped layer and then reshape the outputs back to sequences. We get a validation MSE of about 0.006, which is 25% better than the previous model.  Unstable Gradients Problem To deal with the unstable gradients problem, we can reuse the same tricks for deep nets: » good parameter initialization, faster optimizers, dropout, and so on. However, unlike deep nets such as CNNs, we should not use nonsaturating activation functions (e.g., ReLU) for RNNs. » They may actually lead the RNN to be even more unstable during training. § Small increase in the outputs will eventually cause the outputs to explode after many time steps. » Hence, use a saturating activation function like the hyperbolic tangent. The gradients themselves can explode too. » If you notice that training is unstable, you may want to monitor the size of the gradients (e.g., using TensorBoard) » Use Gradient Clipping when needed.  Layer Normalization Batch Normalization (BN) cannot be used as efficiently with RNNs as with deep feedforward nets. » In fact, you cannot use BN between time steps, only between recurrent layers. » The use of BNs was slightly better than nothing when applied between recurrent layers (i.e., vertically in Figure 15-7), but not within recurrent layers (i.e., horizontally). Layer Normalization: instead of normalizing across the batch dimension, it normalizes across the features dimension. » Like BN, Layer Normalization learns a scale and an offset parameter for each input. » In an RNN, it is typically used right after the linear combination of the inputs and the hidden states. One advantage is that it can compute the required statistics on the fly, at each time step, independently for each instance. » This also means that it behaves the same way during training and testing (as opposed to BN). » It does not need to use exponential moving averages to estimate the feature statistics across all instances in the training set. Implementation of Layer Normalization Use tf.keras to implement Layer Normalization within a simple memory cell. To add dropout, all recurrent layers (except for keras.layers.RNN) and all cells provided by Keras have a dropout hyperparameter and a recurrent_dropout hyperparameter. » The former defines the dropout rate to apply to the inputs (at each time step), and the latter defines the dropout rate for the hidden states (also at each time step).  Long Short-Term Memory In RNNs, some information is lost at each time step. » After a while, the RNN’s state contains virtually no trace of the first inputs. To tackle this problem, various types of cells with long-term memory have been introduced. » They have proven so successful that the basic cells are not used much anymore. The most popular long-term memory cells: the Long Short-Term Memory (LSTM) cell. Two ways to add LSTM layers to a model in Keras: The LSTM layer uses an optimized implementation when running on a GPU, so in general it is preferable to use it.  The Architecture of LSTM Cells The state of a LSTM cell is split into two vectors: h(t) and c(t) » You can think of h(t) as the short-term state and c(t) as the long-term state. The key idea is that the network can learn what to store in the long-term state, what to throw away, and what to read from it. » c(t–1) first goes through a forget gate, dropping some memories, and then adding some new memories (selected by an input gate) via the addition operation. » c(t–1) is copied and passed through the tanh function, and then the result is filtered by the output gate to produce the short-term state h(t), which is equal to the cell’s output for this time step, y(t).  Gates in LSTM Cells The current input vector x(t) and the previous short-term state h(t–1) are fed to four different fully connected layers. The main layer is the one that outputs g(t) given the current inputs x(t) and the previous (short-term) state h(t–1). » Depending on the input gate, g(t) may or may not be added to the long-term state c(t). The three other layers are gate controllers. » The forget gate (controlled by f(t)) controls which parts of the long-term state should be erased. » The input gate (controlled by i(t)) controls which parts of g(t) should be added to the long-term state. » The output gate (controlled by o(t)) controls which parts of the long-term state should be read and output at this time step, both to h(t) and to y(t). Gate controllers use the logistic activation function, whose range is between 0 to 1. » If they output 0s they close the gate, and if they output 1s they open it.  The Equations of LSTM Cells A LSTM cell can be implemented by the following equations: Wxi, Wxf, Wxo, Wxg are the weight matrices of each of the four layers for their connection to the input vector x(t). Whi, Whf, Who, and Whg are the weight matrices of each of the four layers for their connection to the previous short-term state h(t–1). bi, bf, bo, and bg are the bias terms for each of the four layers. Note that Tensor-Flow initializes bf to a vector full of 1s instead of 0s. This prevents forgetting everything at the beginning of training.  Peephole Connections In a regular LSTM cell, the gate controllers can look only at the input x(t) and the previous short-term state h(t–1). An LSTM variant with extra connections called peephole connections » The previous long-term state c(t–1) is added as an input to the controllers of the forget gate and the input gate » The current long-term state c(t) is added as input to the controller of the output gate. Peephole Connections often improve performance, but not always. Keras offers an experimental implementation of LSTM cells with peephole connections » tf.keras.experimental.PeepholeLSTMCell » You can create a keras.layers.RNN layer and pass a PeepholeLSTM Cell to its constructor.  Gated Recurrent Unit (GRU) Cell The GRU cell is a simplified version of the LSTM cell, and it seems to perform just as well » Both state vectors are merged into a single vector h(t). » A single gate controller z(t) controls both the forget gate and the input gate. § If the gate controller outputs a 1, the forget gate is open (= 1) and the input gate is closed (1 – 1 = 0). § If it outputs a 0, the opposite happens. » There is no output gate; the full state vector is output at every time step. § However, there is a new gate controller r(t) that controls which part of the previous state will be shown to the main layer (g(t)).  Equations for GRU Cells The equations for GRU cells: Keras provides a keras.layers.GRU layer. Using 1D convolutional layers to process sequences LSTM and GRU cells are one of the main reasons behind the success of RNNs. » But they still have a fairly limited short-term memory. § They have a hard time learning long-term patterns in sequences of 100 time steps or more, such as audio samples, long time series, or long sentences. One way to solve this is to shorten the input sequences, for example using 1D convolutional layers. » Build a neural network composed of a mix of recurrent layers and 1D convolutional layers (or even 1D pooling layers). » The 1D convolutional layers downsample the input sequence by a factor of 2, using a stride of 2. By shortening the sequences, the convolutional layer may help the GRU layers detect longer patterns.  WaveNet It is possible to use only 1D convolutional layers and drop the recurrent layers entirely. WaveNet stacks 1D convolutional layers, doubling the dilation rate (how spread apart each neuron’s inputs are) at every layer. » The lower layers learn short-term patterns, while the higher layers learn long-term patterns. » Thanks to the doubling dilation rate, the network can process extremely large sequences very efficiently.  WaveNet (cont.) Here is how to implement a simplified WaveNet in Keras: This Sequential model starts with an explicit input layer, then continues with a 1D convolutional layer using "causal" padding. » This ensures that the convolutional layer does not peek into the future when making predictions. Add similar pairs of layers using growing dilation rates: 1, 2, 4, 8, and again 1, 2, 4, 8. Finally, we add the output layer: a convolutional layer with 10 filters of size 1 and without any activation function. GRU with 1D convolutional layers and WaveNet offer the best performance so far in forecasting our time series.