NLP Quiz: Handling Unknown Words and Out-of-Vocabulary Terms

Study Notes

In NLP, an "unknown" refers to a word, phrase, or token that is not present in the training data or vocabulary of a model.
Unknowns can be out-of-vocabulary (OOV) words, special characters, or tokens that are not recognized by the model.

Out-of-vocabulary (OOV) words: Words that are not present in the training data or vocabulary of a model.
Unseen words: Words that are not seen during training but may be seen during testing or deployment.
Special tokens: Tokens that are not part of the standard language, such as emojis, hashtags, or URLs.

Vocabulary mismatch: Models may not be able to handle unknowns, leading to errors or misclassifications.
Overfitting: Models may overfit to the training data, failing to generalize to unknowns.
Lack of robustness: Models may be brittle and fail to perform well when encountering unknowns.

Subwording: Breaking down unknown words into subwords or character-level representations to improve model performance.
Character-level models: Models that operate on character-level representations, rather than word-level representations.
UNK token: Representing unknowns with a special "UNK" token, allowing the model to learn a representation for unknowns.
Vocabulary expansion: Expanding the vocabulary of a model to include more words, reducing the likelihood of unknowns.

Robustness: Handling unknowns improves the robustness of NLP models, enabling them to perform well in real-world scenarios.
Generalization: Models that can handle unknowns are better able to generalize to new, unseen data.
Real-world applications: Handling unknowns is crucial in real-world applications, such as language translation, text classification, and chatbots.