Error Analysis in Deep Learning

Error Analysis

• Error analysis involves examining mislabeled examples in the dev set to prioritize algorithm improvements.

Cleaning Incorrectly Labeled Data

• Incorrectly labeled data should be treated as errors during error analysis.
• If the fraction of incorrectly labeled data is high (e.g., >30%), it should be fixed.
• Deep learning algorithms are robust to random errors but not systematic errors (e.g., consistently mislabeling white dogs as cats) in the training set.

Building Systems Iteratively

• When approaching a new problem, begin by quickly building a simple system
  • Set up your dev/test set and evaluation metrics.
  • Build an initial system rapidly.
  • Use bias/variance analysis and error analysis to prioritize subsequent steps.
• Deep diagnosis may be better if you have experience and specialize in certain applications.

Training and Testing on Different Distributions

• When splitting train/dev/test sets with data from different distributions, avoid shuffling all data together.
• Option 2: Dev/test set all are data from the target, train set will be the leftover data.

Bias and Variance with Mismatched Data Distributions

• Differences between training and dev/test data distributions can make it unclear whether errors are due to variance or data mismatch.
• To detect variance or mismatch problems, split a train-dev set from the training set with the same distribution.
  • If there is a reasonable difference between train vs train-dev, it indicates variance. If not, it is a mismatched data problem.

Addressing Data Mismatch

• Ways to address data mismatch include:
  • Performing error analysis to identify mismatched attributes.
  • Making training data similar to dev/test data, for example, through artificial data synthesis.

Transfer Learning

• Transfer learning involves transferring learned knowledge from task A to task B.
• Transfer learning steps:
  • Adapt the model architecture by changing the output layer for the new task or adding new layers.
  • Initialize the new layer's parameters.
  • Train the new model.
• With a small training set, keep previous parameters and only train the changed layers; with a large training set, retrain all model parameters.
• Pre-training refers to the training process with data from the initial task, and fine-tuning refers to the training process with data from the new task.
• Transfer learning is suitable when:
  • Inputs are of the same type (e.g., image to image, audio to audio).
  • The pre-training task has more data than the fine-tuning task.
  • Low-level features learned in task A are helpful for learning task B.
• Transfer learning is frequently used when limited data is available for the target task, but there is a pre-trained model with ample data and similar low-level features.

Multi-Task Learning

• Multi-task learning involves a single neural network performing multiple tasks simultaneously, with each example potentially having multiple labels.
• Multi-task learning still works even when some labels are missing.
• Conditions in which multi-task learning makes sense:
  • Tasks share similar low-level features.
  • The amount of data for each task is relatively similar.
• A sufficiently large neural network can be trained to perform well on all tasks.
• Using one neural network for multiple tasks is more efficient than using isolated neural networks if the single network is large enough.

End-to-End Deep Learning

• End-to-end deep learning directly maps input x to output y, unlike traditional models with processing stages.
• End-to-end deep learning performs better with large datasets; traditional models may be better otherwise.
• In scenarios like face recognition, splitting the task is better than end-to-end when there is not much data, but large data sets exist for each sub-task (recognizing faces, identifying persons).

Deciding Whether to Use End-to-End Deep Learning

• Pros of end-to-end deep learning:
  • Allows the data to speak, learning data representations without human preconceptions.
  • Requires less manual design of components, saving costs and simplifying the system.
• Cons of end-to-end deep learning:
  • May require a large amount of data.
  • Excludes potentially useful hand-designed components, which can be beneficial with small datasets.
• Key questions to consider:
  • Is there sufficient data to learn a function of the required complexity?
  • Use deep learning to learn individual components.
  • Carefully choose the X→Y mapping based on available data for the tasks.