Natural Language Processing - Exam Lecture 0

Tremendous ambiguity, variability, grounding, and sparsity are some of the challenges that make language hard in NLP.

Zipf's Law reveals that few words make up a great share of the text, with the second most frequent word appearing half as much as the first, the third one appearing one fourth as much, and so on.

NLP draws upon Human & Social Sciences, Linguistics, and Cognition & Neuroscience for its foundation.

The variability of language, including semantic change over years, presents a challenge in understanding the meaning of words and expressions.

Grounding in common sense is essential, as humans don't learn language by observing an endless stream of text but rely on common sense to understand and interpret language.

Extractive Summarization interprets input and generates summary in a more human-like manner, while Abstractive Summarization selects a subset of sentences from the input text to form a summary, typically the most informative sentences from the original.

The objective of text normalization is to harmonize written forms of words with the same meaning, standardize text for easier processing, and reduce language fluctuation.

Numbers and dates are usually replaced by special tokens in NLP preprocessing to prevent the dictionary from exploding, as their inclusion may significantly increase the size of the dictionary.

The purpose of stemming in NLP is to reduce words to their stems by crude affix chopping, aiming to capture the core meaning of words.

Lemmatization uses knowledge resources like WordNet to find and replace base forms, reducing inflectional/variant forms to base forms, while stemming typically involves faster reduction of variation but may harm precision and increase ambiguity.