Language Structure: Words & Morphology

Understanding Speech and Language

• Speech vocalizes language, turning thoughts into sounds.
• Sound in speech has measurable aspects like wave frequency and audible aspects like voice pitch.

Signal and Morphology

• Speech isn't random noise but comprises specific sounds forming words.
• Morphology studies how sounds or morphemes combine to create meaning.

The Structure of Words

• Words are language's building blocks, divisible into smaller units called morphemes.
• Free morphemes can stand alone and make sense, such as "book."
• Bound morphemes need attachment to free morphemes to add meaning, like "-s" making "books" plural.

How Language Changes and Grows

• Open class words readily incorporate new terms, especially from tech and culture like "selfie" or "blog."
• Closed class words like "and," "but," "the" are more resistant to change and new additions.
• Stopwords (e.g., "and", "the") structure sentences but can be crucial for sentiment detection.

The Science of Words in NLP

• Root, stem, and base relate to a word's core meaning.
• Stemming removes "packaging" from a word (e.g., "un-", "-ness" from "unhappiness") to get the core but may remove too much
• Lemma is the dictionary form of a word; it provides a standard reference.

NLP Challenges

• NLP struggles to understand context because the same word can have different meanings.

Inflection

• Inflection modifies words to express grammatical categories like tense or number
• Adding "-ed" converts "walk" to "walked", indicating past tense without altering core meaning.

Derivation

• Derivation creates new words by adding prefixes or suffixes, changing meaning or part of speech
• Adding "-ment" to "employ" yields "employment" with a new meaning and role.

Compounding

• Compounding combines words to create new terms with meanings beyond the sum of parts
• Pronunciation speed can distinguish compound words from phrases.

Combination of Inflection, Derivation, and Compounding

• The combination of compounding ("aircraft") and inflection (adding "-s" for plural) create a term for naval ship designed to carry aircraft.

Homophony

• Homophony refers to words with the same pronunciation but different meanings
• "to," "two," and "too" are English homophones that sound identical but have distinct meanings.

Hyponymy

• Hyponymy is when one word's meaning is included in another's (e.g., "rose" is a hyponym of "flower").

Meronymy

• Meronymy signifies a part-whole relationship; "wheel" is a meronym of "car."

Synonymy

• Synonymy refers to words with nearly identical meanings, often interchangeable, like "big" and "large."

Morphological Richness vs. Poverty

• This describes the extent a language uses morphological processes for meaning.
• Morphologically poor languages (e.g., English) rely on word order and auxiliary words over inflection.
• Morphologically rich languages (e.g., Finnish, Turkish) express more through inflection.

Zipfian Distribution & the Power Law Distribution of Language

• Zipf's Law: any word's frequency is inversely proportional to its rank in a frequency table.
• Distribution Formula: Frequency of the nth word = Frequency of the top word / n.

Characteristics of Frequent Words

• Frequent words tend to be shorter, function words less meaningful in isolation.

Implications of Zipf's Law

• Suggests natural languages optimize for efficiency with high-frequency, shorter words to reduce the effort needed for communication
• Influences algorithms for text summarization, keyword extraction, and language modeling in NLP and text analysis.
• Found outside language in city populations and Internet traffic, showing resource distribution principles.

Part-of-Speech Tagging

• Marks words in a corpus corresponded to a specific part of speech, based on definition and context.

How does POS Tagging work?

• Text broken down into individual words or tokens
• Tokens areassigned POS tags based on context
• Algorithms uses hand-written rules or machine learning on large datasets to determine the correct tag, utilizing contextual information such as surrounding words.

Syntax: The Structure of Sentences

• Syntax are the rules, principles, and processes that govern sentence structure in a given language.
• Arranging the words in a sentence matter significantly for the sentence's meaning.

Phrase: Building Blocks of Syntax

• Phrases have a head word that determines their category and modifier.
• Nested phrases create complexity to express detailed ideas within sentences.

Dependency Relations

• They represent sentence structure by highlighting the relationships between words

Syntactic Tree

• It visualizes sentences structure with part-of-speech and head-modifier relations.

Importance of Verb and Expectations in Sentences

• Verbs set expectations for other sentence elements; their absence leaves a feeling of incompleteness.

Parsers

• Parsers help understand the structure and meaning of sentences in linguistics and NLP.

Context-Free Grammars (CFGs)

• CFGs recursively create patterns of strings
• Symbols replaced with other symbols or terminal elements creates structured sentence representation

Treebanks and Parser Training

• Treebanks are annotated databases with sentences and syntactic structures represented as parse trees
• Parsers trained on treebanks recognize patterns and probabilities in real-world text

Challenges for Parsers

• Real-world parsing is challenged by ambiguity and context-dependent interpretation
• Parses challenged by language like typo, slang, and ungrammatical constructions.

Semantic Parsing and Relations

• Semantic parsing interprets the meaning of sentences beyond grammatical structure.
• Semantic parsing is achieved by mapping sentences to logical representations of entities in a sentence.
• Other semantic roles also includes the theme, beneficiary/recipient, source, path, and goal,

"the man"

• NP[phrase] indicates "the man" is a Noun Phrase with "man" as the noun
• subject[dependency]: In grammar, "the man" is the subject of the sentence
• agent[role]: Semantically, "the man" is the agent doing the opening action.

"the door"

• NP[phrase]: "the door" includes noun "door" with determiner "the".
• direct object[dependency]: "The door" is the recipient of the subject's action.
• patient[role]: Sentence is being opened by "the man".

with a key

• PP[phrase]: "with a key" serves as a Prepositional Phrase, beginning with a preposition and its object.
• adjunct[dependency]: "with a key" functions as an adjunct adding extra information to the verb or action.
• instrument[role]: "With a key" specifies the tool to open "the door."

Pragmatics

• Pragmatics examines how context impacts language interpretation beyond literal meanings.

Metonymy

• Metonymy is a figure of speech where a thing or concept is referred to by closely associated name.

Indirect Requests

• Indirect requests are where sentence form doesn't match intended function or meaning

Words in Semantic Space

• Words exist in a semantic network of meanings and associations, for instance "star".
• Determining correct meaning requires weighting words based on context.

BERT

• BERT improved tasks like question answering, inference, and entity recognition.

Transformer Components

• Tokenizer (T) breaks input text into manageable pieces or tokens for processing.
• Vocabulary (V) is the fixed list of tokens (words) the model understands which are converted numerically.
• Model (M) is the core of BERT that has layers self-attention to process the tokens simultaneously,

What Determines the Semantic Representation of Words in Sentences?

• Tokenizers determine to how text is broken down by boundaries affecting the meaning of phrases.
• After tokenization, each is mapped to high dimensional vector that capture semantic properties of words based on context.
• The self-attention mechanism allows each token to interact with every other token in input the sentence to understand the context of each word.

Contextual Language Models

• Contextual language models predict word meaning via the specific context, unlike a single fixed representation.

XLM-Roberta token vocabulary

• XLM-ROBERTa extends RoBERTa, trained on large corpus of text spanning multiple language.
• XLM-ROBERTa offers advantages for global communication and access across multiple language.

Multilingual Blessing & Curse

• More languages increase performance by cross-lingual similarities
• Too many languages degrade performance because the model capacity is finite
• Capacity & vocabulary can mitigate degradation caused by adding many languages, improving multilingual performance.
• Large, diverse training data for each language included enhances language performance

GPT3's multilingual skills

• GPT-3 exhibits multilingual capabilities, performing tasks in languages it was not explicitly trained on.

Zero-Shot transfer

• Zero-shot transfer applies models to languages/tasks not explicitly prepared for, demonstrating knowledge generalization.