Lecture 3 - Structured Inputs

Lecture 3 - Structured Inputs

• Lecture date and time: November 9, 2023, 10:05 AM

Part I - Designing Features

• Aim: Encode input objects (like text, images, DNA) into numerical values to distinguish them.
• Focus on features that highlight key differences between various objects.
• Objects with rich representations are ideal targets.
• Examples: text documents, images, DNA sequences, molecules.
• Importance is placed on converting sequences (like text or DNA) into understandable numerical formats.

1.1 Encoding Sequences

• Text Sequences:
  • Computers do not interpret words as humans do.
  • Therefore, numerical representation for words in text sequences is needed.
  • Steps in conversion: Tokenization, Vocabulary Building, and Numerical Encoding of Texts.
  • Tokenization: Splitting text into individual words (tokens).
  • Build a vocabulary: Collecting all unique tokens (words) in the dataset.
  • Numerical Encoding: Mapping each token to a unique index from the vocabulary in the initial text.
• Importance of Representations that preserve length: Lengths of text inputs vary, so representations must be the same dimension in feature space. Key is consistent dimensionality for the features.
• Various Different approaches for encoding texts includes:
  • Bag-of-words (Each token is counted)
  • Zero-one
  • Normalized Frequency

1.2 Preserving Sequential Order

• N-grams: Methods that preserve consecutive tokens/words
  • Bigrams: Consider pairs of consecutive words.
  • Trigrams: Consider groups of three consecutive words.
• Static Features Include:
  • Measurements of complexity: Avg characters/word, avg words/sentence (Correlate with readability).
• Other Features: Length (Length Features),lexicon counting (external words counts found frequently).

1.3 Encoding Graphs

• Represent each atom as a node.
• Connect atom-atom bounds using undirected edges.
• Maps bigrams to graphs by visualizing adjacent nodes.
• Graphs represent n-grams in generalized forms.
• Leads to combinations of node numbers that expand on consecutive n-grams.

1.4 Encoding Trees

• Special cases of graphs.
• Tree structure excludes cycles, creating unique advantages for encoding.
• Uses in Modeling: modeling internet conversations, where texts are already structured.
• Uses "prompt/response" pair features to further structure conversational exchanges which is useful.
  • (e.g., In studies of online discussions)

1.5 Encoding Grids

• Grids/Images represented as 3-dimensional tensors.
• Pixels represented by vectors/values, often indicating RGB values.
• Image-patch analysis (neighbouring pixel blocks).
• Feature analysis: Applying filters and analyzing their results for relevant features can improve image analysis.
  • mathematical treatment of images as flattened 3 dimensional tensors, allow for mathematical operations (e.g.inner products).
• Additional Features: edges/corners/other visual elements may be extracted for better image analysis

This lecture focuses on the encoding of various input objects, such as text, images, and DNA, into numerical representations. It emphasizes the importance of feature design to highlight key differences and discusses methods like tokenization and vocabulary building for text sequences. Understand how these processes are crucial for data analysis and interpretation.