Word Embedding, LLM Safety, and RLHF

Word Embedding Recap

• Word embedding is a popular way to represent document vocabulary
• Captures semantic and syntactic similarity, as well as the relationship between words
• Numerical representation of words, where similar words have mathematically similar embeddings and dissimilar words have mathematically dissimilar embeddings
• Tokens are the basic units of a sentence; words, character sets, etc

LLM Safety Concerns

• LLMs possess extensive knowledge, both good and bad
• Pretraining lacks human oversight
• Training data not actively filtered
• Lack of censorship can lead to hazardous and harmful knowledge
• Sensitive data may leak if it’s present in the training data

RLHF - Aligning LLMs

• Reinforcement Learning from Human Feedback is a reinforcement learning approach
• Utilizes human input to fine-tune LLMs
• RL is a machine learning area focused on maximizing cumulative rewards in situations through decision-making
• Learning occurs through experience, unlike supervised methods that rely on predefined datasets
• Agents in RL learn in complex environments by performing actions and adjusting based on rewards or penalties
• Can reduce biased and harmful responses from LLMs, though not perfectly
• Safety mechanisms could be bypassed
• LLM developers must update safety policies

GPT Architecture & Self-Attention

• GPT adopts a "self-attention" mechanism
• Helps in capturing the semantic meaning

Generative Language Models

• Generative model guesses the next word in a series
• Model generates the next word as if it didn't exist
• The process is "auto-regressive generation" as it generates the next word based on all the previous ones
• This model can go beyond the knowledge of people
• It can make decisions from a vocabulary of human languages
• It acquires knowledge at an unprecedented rate, learning patterns without labelled data
• The process is not controlled by humans

LLM Limitations - Hallucinations

• Hallucinations in LLMs: generates inaccurate, fabricated, inconsistent, or nonsensical information
• In-context hallucination: LLM misinterprets input, leading to distorted responses
• Extrinsic hallucination: LLM creates false information due to lack of knowledge

LLM Limitations - Knowledge Timeliness

• LLMs' information can be outdated
• LLMs cannot access information or incorporate data after training

LLM Limitations - Numbers

• LLMs have trouble with numerical comparisons
• Numbers inside LLMs are tokenized

Hyperparameters in LLMs

• Max Tokens: Text length the model generates in one run
• Setting max tokens too low may result in incomplete output
• Setting max tokens too high may include unnecessary information
• Previous Messages Included: Number of previous conversation messages the model remembers
• High value may lead to redundant output and faster burning of your tokens
• Low value may cause forgetting important information from past conversations
• Lower temperatures leverage learned patterns
• Produces predictable and reliable outputs
• Higher temperatures encourage exploration, increasing the diversity of outputs

Prompt Engineering: Query Crafting

• Improves output quality
• Phrasing queries and including context

Prompt Engineering: In-Context Learning

• In-Context Learning (ICL) uses input-output examples in the model's context as a preamble for performing a task on an unseen example
• Zero-shot prompting: No example given
• One-shot prompting: One example provided
• Few-shot prompting: Typically 3~6 examples
• Techniques limited by length limits

Prompt Engineering: Chain-of-Thought

• A sequence of short sentences describes the reasoning, eventually leading to the final answer
• CoT prompting is beneficial for complicated reasoning tasks, particularly with large models

Prompt Engineering: Role Play

• "Role play": uses personas and scenarios
• AI is your actor
• Specific scenarios are constructed through pretend roles
• Improves GPT into a high-moral role

Prompt Engineering: RAG

• Retrieval-Augmented Generation (RAG) combines LLMs with external databases
• Ensure generated content is based on factual data when generating text

Prompt Engineering; Retrievel Augmented Generation example

• Question: Who was awarded the 2024 Nobel Prize in Physics and describe their contributions?
• The model must rely on its trained knowledge
• Models trained on data until September 2024 may not know about the October 2024 Nobel Prize in Physics
• Without RAG: The answer is incorrect
• Without RAG: Admission of Ignorance, or knowledge is outdated
• Model RAG: System retrieves relevant documents, searching keywords like "2024 Nobel Prize in Physics winner"
• With RAG: Retrieved information becomes context, replacing pre-trained knowledge to generate response
• With RAG: Output produces researchers and their artificial information

Prompt Engineering Principles

• Principle 1: Clear instructions
• Request brief or expert replies as needed
• Show the required format if the format is important
• **Principle 2: Reference Text"
• Language models can make up responses especially if asked about niche subjects.
• Reference texts can help prevent the models from fabricating information
• Principle 3: Split complex tasks
• Split complex tasks into simpler sub-tasks
• **Principle 4: External Tools"
• Compensate for the weaknesses of the model by feeding it the outputs of other tools.