AI Fundamentals and Perceptrons

NLP, Text Mining, and Semantic Analysis

• This is a compulsory subject at the IE School of Science and Technology for the 2024/25 academic year.
• The presenter is Alex Martínez-Mingo.

Session II: The Dawn of Computational Linguistics

• This session focuses on the origins of computational linguistics.

What is Computational Linguistics?

• Computational linguistics studies human language using automated computational methods.
• These methods analyze, interpret, and generate human language.

Early Stages and Foundational Theories

• The field of computational linguistics originated in the 1950s, spurred by the advancement of modern computers.
• Previous events also contributed to its development.

The Turing Machine

• Invented by Alan Turing in 1936.
• A theoretical computing device that manipulates symbols on tape based on rules.
• A foundational model for computation, capable of simulating any computer algorithm.
• Crucial for the development of NLP.
• Turing's work during WWII on the Enigma machine was a pioneering linguistic computational challenge.

The Artificial Neuron Model

• Proposed by Warren McCulloch and Walter Pitts in 1943.
• A pioneering conceptual model of a neuron with a simple mathematical model.
• Bridged the gap between biological and computational models in cognitive science and neuroscience.
• Introduced the idea of neural networks, a fundamental concept in NLP.
• Modern deep learning techniques, including recurrent neural networks (RNNs) and transformers, are developments built on these early neural network ideas.

The Information Theory

• Developed by Claude Shannon in 1948.
• Introduced concepts like entropy, information content, and redundancy within communication systems.
• Marked the beginning of digital communication.
• Changed understanding of language as a form of information transfer.
• Enabled quantification of information in language, facilitating NLP analysis.

The N-Gram Model

• Shannon's entropy concept is crucial for language modeling.
• The goal of language modeling is predicting sequence probabilities of words.
• N-grams are a practical application of information theory to language modeling.
• N-gram models predict the probability of a word based on the occurrence of the preceding (N-1) words.
• This approach is a form of the Markov model (1913).

Early Stages and Foundational Theories (Georgetown Experiment)

• The Georgetown Experiment (1954) is one of the earliest applications of n-grams.
• Automated translation of Russian to English, using approximately 250 words and six grammatical rules.
• Successfully translated 60 sentences.

The Perceptron

• Developed by Frank Rosenblatt in 1958.
• An early model in artificial intelligence.
• Mimicked the human brain's decision-making process.
• Operated by weighing input signals, summing them, and processing via a non-linear function to produce an output.
• Provided a fundamental model for how machines process and classify linguistic data.
• Essential concepts from perceptrons remain relevant in current NLP methodologies.

The Linguistic Wars

• Significant intellectual debate within 20th-century linguistics.
• Primarily between generative (Noam Chomsky) and behaviorist (B.F. Skinner) linguists, regarding language nature, acquisition, & understanding.
• No clear winner; Chomsky's and Universal Grammar theories impacted linguistic theory.
• Empirical and cognitive frameworks added to this discourse.

The Multi-Layer Perceptron

• Proposed by Marvin Minsky and Seymour Papert as an extension of Rosenblatt's perceptron.
• Learned complex patterns by combining outputs from previous layers.
• Influenced the development of further neural network architectures in advanced NLP tasks.
• A core component in many modern NLP systems.

The XOR Problem

• Single-layer perceptrons are unable to solve problems where data isn't linearly separable.
• Minsky and Papert used XOR (exclusive OR) for critique in "Perceptrons" (1969).
• Led to disillusionment in the AI community and a funding reduction—a significant AI winter.

The First AI Winter

• The 1960s and 1970s were marked by a "golden age" of rule-based NLP.
• NLP during this time was predominantly based on rule-sets and Regular Expressions (RegEx).
• Early AI programs such as ELIZA (1966) and SHRDLU (1972) were some examples of first AI systems.
• Others sought to explain human language algorithmically.

The Semantic Network

• Proposed by M. Ross Quillian in the 1960s.
• Represents knowledge as a graph of interconnected nodes (concepts) connected via links representing relationships.
• Demonstrates enhanced information retrieval using networked structures.
• Influential in the development of knowledge graphs and ontology-based systems in NLP.

The Semantic Memory

• Proposed by Endel Tulving in the 1970s.
• Is a system for storing general knowledge of the world, rather than personal experiences (unlike episodic memory).
• Provides a theoretical basis for understanding how knowledge and language are stored & retrieved in the human brain.
• Guides the design of knowledge-representation systems within NLP.

The Prototype Theory

• Developed by Eleanor Rosch in the 1970s.
• Challenges the classical categorization theory.
• Proposes that categories are centered on prototypes or typical examples instead of necessary & sufficient characteristic sets.
• Prototypes are often the best or most typical instance of a category.
• Influenced understandings of concept organization and categorization, influencing categorization and clustering algorithms in NLP.

The Renaissance of Connectionist Models

• A resurgence of connectionist models in the 1980s.
• Particularly around 1986, marked a significant shift in the field's perspectives and attention.

The Backpropagation Algorithm

• Developed by David Rumelhart, Geoffrey Hinton, and Ronald Williams in 1986.
• Enables efficient training of multi-layer perceptrons (MLPs).
• Adjusts weights not just for output, also across all hidden layers.
• Learns complex patterns & non-linear separations (like the XOR problem).
• Employs forward and backward propagation steps for error calculation & weight updates.

Feedforward Models

• Enabled by backpropagation algorithms.
• Neural networks with non-cyclic connections.
• Used for NLP classification & regressions tasks.
• Advantages over n-gram models:
  • Capture more complex language patterns.
  • More flexible context sizes, reducing data sparsity issues for generalization.
• Limitations:
  • Struggle with capturing long-term dependencies in sequential data, needing internal memory for prior inputs for future predictions.

Recurrent Neural Networks (RNNs)

• Developed by Jeffrey Elman in 1990.
• Designed to process sequences by maintaining internal state (memory).
• Ideal for sequential data requiring order awareness and contextual input understanding.
• Crucial in generative language models.
• Limitation: "vanishing gradient" problem.

Recurrent Neural Networks (RNNs) - Vanishing Gradient

• Trained using Backpropagation Through Time (BPTT).
• Unrolls the RNN for long sequences, leading to deep networks.
• Gradients are propagated backward through time and multiplied via weight matrices at each step during backpropagation.
• Weight multiplications lead to a diminishing gradient "vanishing" in effect.
• Makes it difficult for RNNs to learn and retain information from the earlier steps within the sequence.

The Second AI Winter

• AI advancements failed to meet expectations and requirements in the 1980s.
• Hardware restrictions on model complexity and dataset size.
• Decreased funding for and support from investors & governments.
• Led to a shift towards more feasible rule-based and statistical approaches, including corpus-based linguistics.

Corpus-Based Linguistics

• The Brown Corpus and British National Corpus (BNC).
• Started in the 1960s and completed in the 1970s/1980s.
• Becoming publicly available in the 1990s.
• Provided a massive dataset for researchers enabling effective statistical linguistic methods.

Statistical Methods and Machine Learning

• During the second AI winter, statistical methods gained prominence.
• Provided unique approaches for making predictions about text data.
• Subsequent application of machine learning enhanced algorithm performance.

Statistical Methods and Machine Learning - Models

• Naive Bayes (based on Bayes' Theorem) became popular for text classification.
• Utilized extensively for detecting spam emails & categorizing documents.
• Aided by its efficiency in handling large datasets.
• Logistic Regression, an older statistical method, surged in NLP for binary classification.
• Best employed when features (words/phrases) and categories exhibited more linear (and less complex) relationships.

The Geometry of Meaning

• First spatial models of language were developed in the 1990s.
• Latent Semantic Analysis (LSA) model (Deerwester et al., 1990).
• Used a term-document matrix and SVD (Singular Value Decomposition) to represent both terms & documents as vectors.
• Hyperspace Analogue to Language (HAL) model (Lund and Burgess, 1996), used co-occurrence matrices and employed dimensional reduction methods to represent terms in a vector space.

The Geometry of Meaning - Word2Vec

• Mikolov et al. (2013a, 2013b) introduced Word2Vec.
• Two architectures (CBoW and Skip-Gram) to create dense word vectors using neural network models.

The Geometry of Meaning - GloVe

• GloVe was developed by Pennington, Socher, and Manning (2014) at Stanford.
• Explores the use of a co-occurrence matrix between words, across context windows, to encode relationships between words.
• combines aspects of LSA/Matrix factorization and Word2Vec (context-based learning) to provide global word co-occurrence statistics.

The Last Connectionist Wave

• Recent resurgence and refinement of more complex connectionist models in the 2010s and beyond demonstrated high success.

Long-Short Term Memory (LSTM)

• Proposed by Hochreiter & Schmidhuber (1997) to address RNN's "vanishing gradient" problem.
• Uses memory cells capable of retaining long-term information via gate mechanisms (input, output, forget).
• This was essential for understanding and improving generative language models and other sequence-dependent tasks.

Transformers

• Introduced by Vaswani et al. (2017) in the landmark paper "Attention is All You Need".
• Handles long-range dependencies more efficiently through self-attention mechanisms.
• Weights the importance of different parts within input data, irrespective of their position within a sequence.
• Facilitates training parallelization.
• Scaled well with data and computational resources enabling its use in large-scale NLP tasks.

Large Language Models (LLMs)

• Advanced AI models trained on massive text corpora.
• Key to performance is using massive amounts of data and high numbers of model parameters (billions).
• Require significant computational power (GPUs or TPUs) and consume substantial resources.
• Larger models demonstrate better performance due to scaling and improvement in abilities for generalization tasks.

Large Language Models - Training Compute

• Training compute (FLOPs) of notable models.
• Graph visually plots increases over time.

Large Language Models - In-Context Learning and Step-by-Step Reasoning

• In-Context Learning: LLMs excel in context maintenance across extended passages.
• Step-by-Step Reasoning: LLMs can mimic step-by-step problem-solving, logical reasoning, & technical troubleshooting.

Assignment: In-Depth Exploration of NLP Models or Techniques

• Choose one NLP model or technique from the course catalog.
• Independently investigate the chosen topic.
• Research models or topics in detail. Utilize various sources including ChatGPT.
• Write a one-page essay summarizing the model and discussing its development, underlying principles, applications, strengths, limitations, etc.
• Submit the one page essay via Turnitin in PDF format before a designated due date.

This quiz explores the significant contributions of Minsky and Papert on single-layer perceptrons and the broader implications of their work on artificial intelligence. It covers their conclusions, the impact of their book 'Perceptrons', and other essential topics regarding neural networks and semantic memory. Test your understanding of these critical AI concepts!