Machine Learning Yearning Study Notes

A single metric that summarizes a model's performance on a dataset, making it easy to compare different models.

The percentage of correctly classified instances out of all the instances in a dataset.

Measures how many relevant items are selected out of all the selected items.

Measures how many relevant items are selected out of all the relevant items.

A combined metric that considers both Precision and Recall, providing a balanced measure of a model's performance.

A set of data used for training a machine learning model.

A set of data used to evaluate the performance of a trained machine learning model.

Using a single number to quickly gauge a model's overall performance.

The error rate of a machine learning algorithm on a very large training set.

The difference in performance between a machine learning algorithm's training set and its test set.

Improving a machine learning algorithm's performance on the training set.

Making a machine learning algorithm generalize better to unseen data.

The difference in performance between a machine learning algorithm's training set and its development set.

The goal of improving a machine learning model's performance.

The difference in error rate between the training set and the development set.

The error rate of the machine learning algorithm on the training set.

The dataset used to evaluate the performance of a machine learning model during development.

The dataset used to evaluate the final performance of a machine learning model after development.

When a model performs exceptionally well on the dev set but poorly on the test set, it indicates that the model has learned the specific patterns of the dev set too well, making it less generalizable.

A process of repeatedly evaluating models on the dev set, which can lead to overfitting. It's best to use a fresh dev set during development to prevent overfitting.

When a model performs well based on a chosen evaluation metric, but fails to address the actual goals of the project.

The act of choosing a new evaluation metric to accurately measure the model's performance in relation to the project goals.

The process of choosing the best model based on the evaluation metric, but failing to consider other aspects of performance like user experience.

The iterative process of developing and evaluating models, adjusting dev sets, and modifying metrics to ensure the model's performance meets the project's objectives.

A subset of the development set used for manual error analysis to understand the algorithm's weaknesses.

A portion of the development set kept separate from the Eyeball Dev Set for evaluating model performance and tuning hyperparameters without human intervention.

The process of using a development set to understand the types of errors an algorithm is making by examining misclassified examples.

When an algorithm's performance on a specific dev set improves at a disproportionate rate compared to its performance on a separate dev set, suggesting that it's being overly tuned to the first set. This is a sign that the algorithm is memorizing the training data instead of truly understanding the underlying patterns.

The size of the Eyeball Dev Set should be large enough to give you a good understanding of the algorithm's main error categories.

A visual representation of a function or algorithm's performance as a function of the size of the data set. The curve typically flattens out as a model learns more from the data, indicating that it is generalizing well.

The process of adjusting the hyperparameters of a model, such as the learning rate or the number of hidden layers, to improve its performance on the development set.

Parameters are the internal variables of a machine learning model that are adjusted during training. Hyperparameters are settings that control the training process of the model.

A model that performs well on the training data but poorly on the dev set, indicating it has learned the specific patterns of the training data too well and fails to generalize to new data.

A model that performs poorly on both the training and dev sets, suggesting it is not able to learn the underlying patterns in the data.

A model that performs well on both the training and dev sets, showing it has learned the patterns of the data without overfitting.

The total error of a model is the sum of its bias and variance. This means a model's overall performance is influenced by how well it fits the training data and how consistently it performs on new data.

Techniques that aim to simultaneously reduce both bias and variance in a model.

Major changes to the architecture of a model aimed at reducing both bias and variance.

You can get a team of doctors to label data, typically with a 2% error rate.

By discussing images with a team of doctors, you can leverage their intuition for error analysis.

2% error rate is often considered the optimal achievable rate for a team of doctors.

Doctors are expensive, so you might use less experienced doctors for simple cases and reserve the experts for harder ones.

When your performance is already high, using human-level labels as a benchmark becomes crucial for further improvement.

Even when humans have a high error rate, if your system is worse, there's potential for improvement.

Humans often excel in specialized areas where machines struggle, like quickly spoken speech in noisy environments.

Using different subsets of data that highlight human strengths can accelerate AI improvement.