Capitolo 15: Sequenze con RNN e CNN

Chapter 15: Processing Sequences Using RNNs and CNNs

• The chapter focuses on processing sequences using Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs).
• RNNs are a class of networks capable of predicting the future; they analyze time series data, such as stock prices.
• The chapter will cover fundamental RNN concepts, training methods (backpropagation through time), forecasting time series, and issues with short-term memory and unstable gradients.
• A specific CNN architecture, WaveNet, is introduced as capable of processing time series data like RNNs.

Recurrent Neurons and Layers

• Recurrent neural networks are similar to feedforward networks but include connections going backward.
• A recurrent neuron, unrolled through time, receives inputs x(t) and its output from the previous time step Y(t-1).
• At each time step (frame), the neuron receives input x(t) and output from the previous time step Y(t-1).
• In the first time step, the input is 0.

A Layer of Recurrent Neurons

• Every neuron at each time step t receives the input vector x(t) and the output vector from the previous time step y(t-1).
• Each recurrent neuron has two weight sets: one for inputs x(t) (Wx) and one for outputs of the previous time step y(t-1) (Wy).
• For a single instance, the output y(t) is wx x(t) + wy y(t-1) + b.
• For a mini-batch, the output Y(t) = (x(t)Wx + Y(t-1)Wy + b).

Memory Cells

• A recurrent neuron, and a layer of them, has memory because its output depends on all inputs from past time steps.
• A memory cell (or simply a cell) preserves state across time steps.
• A single recurrent neuron/layer can learn shorter patterns; more powerful cells are needed for longer patterns.
• The cell's state (h(t)), at time step t, is a function of inputs at t and previous state h(t-1).
• The output (y(t)) at time step t is also a function of previous state, h(t-1), and current inputs, x(t).

Input and Output Sequences

• Sequence-to-sequence networks: For example, predicting time series data or translating a sentence from one language to another.
• Sequence-to-vector networks: E.g., taking a sequence of words and generating a sentiment score.
• Vector-to-sequence networks: E.g., creating a caption for an image.
• Encoder-Decoder: A type of network where an "encoder" translates an input sequence into a vector, and a "decoder" translates that vector back into an output sequence.

Training RNNs

• Backpropagation Through Time (BPTT): A method for training RNNs that involves a forward pass through the unrolled network, evaluating output sequences against a cost function, propagating gradients backward through the unrolled network, and updating model parameters.
• The cost function in this algorithm can ignore some outputs.
• Parameters are updated using gradients, computed during the BPTT and summed over all time steps.

Forecasting A Time Series

• A time series is data sequenced over time.
• A univariate time series has single value per time step, and multivariate time series has multiple values.
• Forecasting is predicting future values in a series.
• Imputation is predicting missing values from past data.

Generating Time Series for Experiments

• Synthetic time series data is created for experiments instead of real-world stock price data.
• The generated series is the sum of two sine waves and noise.
• The series is a NumPy array of shape [batch size, time steps, 1]

Baseline Metrics

• Baselines are used for performance comparison.
• Naive Forecasting: Predicts the next value using the last value.
• Linear Regression Model: A simple linear model as a baseline, using a fully connected network for comparison.

Implementing a Simple RNN

• A very simple RNN model is built using Keras and compared to baseline methods.
• It consists of a single layer with a single neuron.
• The default activation function is hyperbolic tangent.
• The initial state is set to 0, and the neuron computes a weighted sum, applies the activation function to compute the output.

Trend and Seasonality

• Removing trend and seasonality might be needed in another forecasting models.
• RNNs usually do not require this initial step, as the model has already learned some trends and patterns.

Deep RNNs

• A deep RNN has multiple layers.
• Setting return_sequences to True for all but the last recurrent layer is important.

Forecasting Several Time Steps Ahead

• Methods to predict several values ahead, including predicting next value, and recursively adding that prediction into the input.
• Predicting multiple values at once (sequence to sequence).

WaveNet

• Uses only 1D convolutional layers instead of recurrent layers.
• The dilation rate (spacing of input connections) doubles at each layer, allowing the network to learn long-term patterns efficiently.

Long Short-Term Memory (LSTM)

• RNNs with limited short-term memory.
• LSTM cells preserve their state across multiple steps.
• Two ways to add LSTM layers to a model in Keras are using LSTM or LSTM cells.

The Architecture of LSTM Cells

• LSTM cells have a state composed of a short-term state (h(t)) and a long-term state (c(t)).
• Key components: forget gate, input gate, output gate, and cell state computation for the calculation of the next cell state.

Gates in LSTM Cells

• Four different gates handle information flow (forget, input, output).
• Gate controllers influence the long-term state based on input and the previous cell state.
• The gate controller uses the logistic activation function to control the gate (0 or 1).
• If the output of gate function is 0, it closes the gate; otherwise it opens the gate.

Equations for LSTM Cells

• Equations for computing gate activations, the new cell state, and the output for each time step.

Peekhole Connections

• Peephole Connections are an LSTM variant extending gate controller connections, adding information from previous states to further enhance memory.

Gated Recurrent Unit (GRU) Cell

• GRU cell is a simpler version of the LSTM cell, with fewer gates and performing equally well as LSTM.
• Single gate controller (z(t)) controls both forget and input gates for h(t).
• No output gate, directly outputs values.

Equations for GRU Cells

• The equations for calculating the update gate z(t), the reset gate r(t), the new hidden state g(t), and the final hidden state h(t) using a single gate controller.

Using 1D Convolutional Layers

• 1D convolutional layers are used to process sequences, shortening the input sequences and helping the GRU layers detect longer patterns.

Unstable Gradients Problem

• Unstable gradients are a problem in deep nets and can be solved using techniques like good initialization, better optimizers, dropout and not using nonsaturating activation functions.
• Use of activation functions like the hyperbolic tangent is important.

Layer Normalization

• Layer normalization is used instead of batch normalization.
• It normalizes across the feature dimension, reducing the issue with gradient exploding related to the sequence length issue.
• Layer normalization is beneficial because it computes statistics on the fly for each instance.

Questo capitolo approfondisce l'elaborazione delle sequenze attraverso Reti Neurali Ricorrenti (RNN) e Reti Neurali Convoluzionali (CNN). Verranno trattati concetti fondamentali delle RNN, metodi di addestramento e l'architettura WaveNet per la previsione di dati sequenziali. È ideale per chi desidera comprendere le dinamiche e le applicazioni delle reti neurali nel contesto delle serie temporali.