Deep Neural Networks II - CNNs and RNNs

<p>Recurrent connections with shared weights (D)</p> Signup and view all the answers

<p>Backpropagation through time equation (A)</p> Signup and view all the answers

<p>It relies on implicit labels extracted from data points. (B)</p> Signup and view all the answers

<p>It facilitates the reuse of pretrained models on unrelated tasks. (D)</p> Signup and view all the answers

<p>Defining network architecture for variable-length inputs. (B)</p> Signup and view all the answers

<p>Image recognition (C)</p> Signup and view all the answers

<p>Data augmentation (B)</p> Signup and view all the answers

<p>Availability of labeled training data. (A)</p> Signup and view all the answers

<p>Many CNN architectures and pretrained models are available. (C)</p> Signup and view all the answers

<p>Utilizing historical data for future predictions. (B)</p> Signup and view all the answers

<p>It has hyperparameters like size and stride. (A)</p> Signup and view all the answers

<p>Convolutional layers and pooling layers. (C)</p> Signup and view all the answers

<p>A differentiable function transforms the input volume. (B)</p> Signup and view all the answers

<p>Fully connected layers. (D)</p> Signup and view all the answers

<p>It involves optimizing the learning process using methods like SGD and Adam. (D)</p> Signup and view all the answers

<p>Recurrent layer. (D)</p> Signup and view all the answers

<p>Each layer applies a differentiable function to transform the input volume. (B)</p> Signup and view all the answers

<p>The entire network can be trained simultaneously from input to output. (B)</p> Signup and view all the answers

<p>Short-term memory limitations in standard RNNs (C)</p> Signup and view all the answers

<p>Using LSTMs instead of vanilla RNNs (A)</p> Signup and view all the answers

<p>It determines which information is discarded from the cell state (D)</p> Signup and view all the answers

<p>Gates are used to learn and decide on information flow (D)</p> Signup and view all the answers

<p>Cell states replace the hidden state of standard RNNs (C)</p> Signup and view all the answers

<p>GRUs are redundant compared to LSTMs (C)</p> Signup and view all the answers

<p>$h = o \circ \sigma(c)$ (C)</p> Signup and view all the answers

<p>Gates create parallel pathways for information flow (B)</p> Signup and view all the answers

<p>To represent categorical data as binary vectors (B)</p> Signup and view all the answers

<p>Start of sequence (D)</p> Signup and view all the answers

<p>Information from earlier inputs is often forgotten (D)</p> Signup and view all the answers

<p>By allowing the decoder to focus on relevant hidden states (C)</p> Signup and view all the answers

<p>It varies based on the importance of hidden states (C)</p> Signup and view all the answers

<p>Neural networks or similarity measures (D)</p> Signup and view all the answers

<p>Incorporating attention to leverage all hidden states (B)</p> Signup and view all the answers

<p>They compress sequences into a single embedding (B)</p> Signup and view all the answers

<p>They generate a massive number of parameters. (B)</p> Signup and view all the answers

<p>Spatial locality of pixels. (B)</p> Signup and view all the answers

<p>To compute the dot product with input locations. (D)</p> Signup and view all the answers

<p>Convolutional layers consider local pixel relationships. (A)</p> Signup and view all the answers

<p>It reduces the output feature map size. (B)</p> Signup and view all the answers

<p>To aggregate values and reduce representation size. (A)</p> Signup and view all the answers

<p>They replace matrix multiplication with convolution. (D)</p> Signup and view all the answers

<p>Multiple output feature maps are generated. (B)</p> Signup and view all the answers

<p>To avoid losing edge information. (B)</p> Signup and view all the answers

<p>Any input laid out on a grid (e.g., images, time-series). (D)</p> Signup and view all the answers

<p>Filters learn a hierarchy from lower to higher spatial features. (C)</p> Signup and view all the answers

<p>They control the size of the output feature map and computational efficiency. (C)</p> Signup and view all the answers

<p>They have mechanisms for handling long-range dependencies in sequences. (B)</p> Signup and view all the answers

A specialized neural network for grid-like data like images. It uses convolution instead of general matrix multiplication in at least one layer.

A mathematical operation used in CNNs. It involves taking the dot product of a filter (kernel) and each input location.

A small matrix used in convolution that extracts specific features from the input data (e.g., edges or corners).

The output of a convolutional layer. It represents the activation of a specific feature in an input.

A standard neural network layer that connects every neuron in the previous layer to every neuron in the current layer.

A matrix or vector of numbers (e.g., 0-255) representing an image.

The amount of resources (memory and processing power) needed to perform a calculation.

The arrangement of elements in a grid-like format. It emphasizes the importance of the elements' spatial relationship.

Adding zeros around the border of an input to maintain output size during convolution.

A technique to downsample the input feature maps by summarizing values, such as finding the maximum or average of a group of pixels.

The number of pixels moved when applying a convolution filter.

Images with multiple color components (e.g., RGB).

Output of a single filter application over an image region.

Settings that control the learning process, such as filter size, stride, and padding.

A layer in CNNs that downsamples feature maps by summarizing values (e.g., max or average) within a region. It has no trainable parameters but may have hyperparameters like size and stride.

A sequence of convolutional and pooling layers that extract features from input data, followed by fully connected layers for classification or regression.

A process in CNNs that changes an input 3D volume (like an image) into an output 3D volume using a differentiable function, which may or may not have parameters.

A list of layers that transform an image volume into an output volume, often consisting of convolutional, pooling, and fully connected layers. Each layer applies a differentiable function and may or may not have parameters or hyperparameters.

A method used to train CNNs by calculating the gradients of the loss function and updating the model parameters iteratively.

A standard neural network layer that connects every neuron in the preceding layer to every neuron in the current layer. This allows for complex interactions between features.

Changes or differences in the input data, such as different angles, lighting, or backgrounds. Pooling layers are robust to input variations.

Adjusting the weights of a pre-trained model on a new, smaller dataset that is similar to the original training data.

Using a pre-trained model on a different task by fine-tuning the weights on a new dataset, even if the new task is unrelated to the original one.

Training a model without labeled data by extracting implicit labels from the data itself.

A type of neural network designed to process sequential data, like text or speech, by remembering past information to predict future items.

Identifying the emotional tone in text (e.g., positive, negative, neutral).

Converting spoken language into written form.

Translating text from one language to another.

Creating new text based on learned patterns.

A problem where the gradient signal during backpropagation decreases exponentially as it flows through layers, leading to slow or ineffective learning.

A problem where the gradient signal during backpropagation increases exponentially as it flows through layers, leading to instability and divergence in the learning process.

Recurrent Neural Networks (RNNs) struggle to learn long-term dependencies in sequences due to the vanishing or exploding gradient problem.

The length of a sequence acts like the depth of a neural network in RNNs, impacting gradient flow and learning.

The derivative of the tanh function is always less than 1, causing the gradient to become smaller with each layer, contributing to the vanishing gradient problem.

A way of representing the target output of the decoder. Each output symbol is encoded as a vector with a single '1' at the index corresponding to the symbol, and all other elements are '0'.

A neural network component in seq2seq models responsible for encoding an input sequence into a fixed-length vector, representing the compressed essence of the input.

A neural network component in seq2seq models responsible for generating an output sequence based on the encoded representation of the input sequence.

A mechanism that allows the decoder to focus on specific parts of the encoded input sequence when generating the output sequence. It helps the decoder to access information from the entire input sequence dynamically.

A vector that represents the current state of the decoder. It is dynamically calculated based on the encoded input and the attention mechanism, reflecting the decoder's understanding of the input sequence.

Values computed by the attention mechanism to determine the relative importance of different parts of the input sequence for the decoder.

A method of generating a sequence where each output symbol is predicted based on the previously generated symbols.

The tendency for RNNs to forget information from earlier parts of the input sequence as they process longer sequences.

Vanilla RNNs have trouble learning and remembering long-term dependencies in data due to the vanishing gradient problem, causing them to struggle with sequences where information needs to be preserved over extended time periods.

LSTMs are a type of RNN designed to address the vanishing gradient problem by introducing a memory cell and gating mechanisms. This enables LSTMs to learn and retain long-term dependencies in data more effectively.

Gates are components within LSTMs that control the flow of information within the memory cell. These gates determine which information is retained, forgotten, or used to update the cell state.

In LSTMs, the forget gate is responsible for deciding which information from the previous cell state should be discarded or forgotten. It selectively removes outdated or irrelevant information.

In LSTMs, the input gate determines which new information from the current input is allowed to enter and update the cell state.

In LSTMs, the output gate controls the information flow from the cell state to the hidden state. It decides what information is relevant for the output at the current timestep.