Deep Learning Layer Types

Layer Types

• There are several types of layers in deep learning, including:
  • Core layers (Input, Dense, Activation)
  • Convolution layers (Conv1D, Conv2D, Conv3D)
  • Pooling layers (MaxPooling1D/2D/3D, AveragePooling1D/2D/3D, GlobalMaxPooling1D/2D/3D)
  • Reshaping layers (Reshape, Flatten, Cropping1D/2D/3D, UpSampling1D/2D/3D, ZeroPadding1D/2D/3D)
  • Merging layers (Concatenate, Average, Maximum, Minimum)
  • Normalization layers (BatchNormalization, LayerNormalization)
  • Regularization layers (Dropout, SpatialDropout1D/2D/3D, GaussianDropout, GaussianNoise)

Data Normalization

• Normalization involves changing the range of input values to:
  • [0,1]
  • [-1,1]
  • mean=0, std_dev=1
• Normalization stabilizes the model's behavior during training and speeds up training
• The training process involves normalizing inputs and outputs, training the model with normalized inputs and outputs
• The inference process involves normalizing inputs, running them through the model to get normalized outputs, and then denormalizing the outputs

Normalization Layers

• Normalization can be done within the network using:
  • LayerNormalization (normalizes activations of the previous layer for each example)
  • BatchNormalization (normalizes activations of the previous layer across a batch)
• Normalization norms include L1 and L2

Dropout

• Dropout is a regularization strategy that randomly cancels features during training
• Dropout helps the network avoid overfitting and learn in a more generic way
• SpatialDropout1D/2D/3D drops entire feature maps in 1D, 2D, and 3D
• GaussianDropout multiplies with 1-centered Gaussian noise
• GaussianNoise adds 0-centered Gaussian noise

Loss Functions

• Loss functions include:
  • Probabilistic losses
  • Regression losses
  • Hinge losses for "maximum-margin" classification

Probabilistic Losses

• Binary Cross-entropy (log-loss, binary problems) is defined as:
  • −(1/N) ∑ (ygt.log(ypred)+(1−ygt).log(1−ypred))
• Categorical Cross-entropy (log-loss, multiple classes, one-hot representation) is defined as:
  • −(1/N) ∑ ygt.log(ypred)
• Sparse Categorical Cross-entropy (log-loss, multiple classes, labels provided as integers) is defined as:
  • Shape of ygt is [batch_size], shape of ypred is [batch_size, num_classes]

Examples

• Binary Cross-entropy example:
  • BCEL = -(1/5) ( [0 0 1 1 0].log2 [0+ε 0+ε 1-ε 1-ε] + [1 1 0 0 1].log2 [1-ε 1-ε 0+ε 0+ε] )
  • BCEL = -(1/5) ( [0 0 -16 0 0] + [0 0 0 0 -16] ) = -1/5 (-32) = 6.4
• Categorical Cross-entropy example:
  • CCEL = -1/3 ([0 0 1 0].log[0+ε 0+ε 1 0+ε] + [1 0 0 0].log[0.9 0+ε 0.1 0+ε] + [0 1 0 0].log[0.2 0.3 0.5 0+ε])

Layer Types

• There are several types of layers in deep learning: Core, Convolution, Pooling, Reshaping, Merging, Normalization, and Regularization
• Core layers include Input, Dense, and Activation layers
• Convolution layers include Conv1D, Conv2D, and Conv3D
• Pooling layers include MaxPooling1D/2D/3D, AveragePooling1D/2D/3D, and GlobalMaxPooling1D/2D/3D
• Reshaping layers include Reshape, Flatten, Cropping1D/2D/3D, UpSampling1D/2D/3D, and ZeroPadding1D/2D/3D
• Merging layers include Concatenate, Average, Maximum, and Minimum
• Normalization layers include BatchNormalization and LayerNormalization
• Regularization layers include Dropout, SpatialDropout1D/2D/3D, GaussianDropout, and GaussianNoise

Data Normalization

• Normalization is the process of changing the range of input values to [0,1], [-1,1], or mean=0, std_dev=1
• Normalization stabilizes the model's behavior during training and speeds up training
• The training process involves normalizing inputs and outputs, training the model with normalized inputs and outputs, and then denormalizing the outputs
• The inference process involves normalizing inputs, running them through the model to get normalized outputs, and then denormalizing the outputs

Normalization Layers

• Normalization can be done within the network using LayerNormalization or BatchNormalization
• LayerNormalization normalizes the activations of the previous layer for each example, maintaining a mean activation close to 0 and a standard deviation close to 1
• BatchNormalization normalizes the activations of the previous layer across a batch, maintaining a mean output close to 0 and a standard deviation close to 1
• Normalization norms include L1 and L2

Dropout

• Dropout is a regularization strategy that randomly cancels features during training, forcing the network to learn in a more generic way
• Dropout helps the network avoid overfitting
• SpatialDropout1D/2D/3D drops entire feature maps in 1D, 2D, or 3D
• GaussianDropout multiplies with 1-centered Gaussian noise
• GaussianNoise adds 0-centered Gaussian noise

Loss Functions

• There are three types of loss functions: probabilistic, regression, and hinge losses
• Probabilistic losses include Binary Cross-entropy and Categorical Cross-entropy
• Binary Cross-entropy is used for binary problems, while Categorical Cross-entropy is used for multiple classes with one-hot representation or sparse labels

Probabilistic Losses

• Binary Cross-entropy calculates the loss as -(1/N) Σ (ygt.log(ypred) + (1-ygt).log(1-ypred))
• Categorical Cross-entropy calculates the loss as -(1/N) Σ ygt.log(ypred)
• Sparse Categorical Cross-entropy calculates the loss as -(1/N) Σ ygt.log(ypred), where ygt is a sparse label and ypred is a probability distribution