Evaluating LLM-Generated Content

Evaluating LLM-Generated Content

• Evaluation methods measure LLM performance. Manual evaluation is time-consuming and costly, so automatic evaluation is common.
• Automatic methods assess qualities like fluency, coherence; relevance, factual consistency, and fairness, and similarity to a reference.

Reference-Based Metrics

• Reference-based metrics compare generated text to a human-annotated "ground truth" text. Many metrics were developed for traditional NLP tasks, but apply to LLMs.
• N-gram based metrics (BLEU, ROUGE, JS divergence) measure overlap in n-grams between the generated and reference texts.

BLEU (Bilingual Evaluation Understudy)

• Used to evaluate machine-translated text, also applicable to other tasks like text generation.
• Measures precision, which is the fraction of candidate words in the reference.
• Scores are calculated for segments (often sentences), then averaged for the whole corpus.
• Doesn't account for punctuation or grammar.
• A perfect score (candidate identical to a reference) isn't needed; more than one reference is helpful.
• Formula to find Precision (P): P = m/wt, where m = number of candidate words in reference, wt= total number of words in candidate.
• Can calculate for n-grams (bi-grams, tri-grams) to improve accuracy.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

• Measures recall, useful for evaluating generated text and summarization.
• More focused on recalling words from the reference.

ROUGE-N

• Measures matching n-grams between reference (a) and test (b) strings.
• Precision: (number of n-grams in both a and b) / (number of n-grams in b)
• Recall: (number of n-grams in both a and b) / (number of n-grams in a)
• F1 score: (2 * precision) / recall

ROUGE-L

• Measures the longest common subsequence (LCS) between reference (a) and test (b) strings.
• Precision: LCS(a,b) / (number of unigrams in b)
• Recall: LCS(a,b) / (number of unigrams in a)
• F1 score: (2 * precision) / recall

Text Similarity Metrics

• Text similarity metrics compare overlap.
• Useful for evaluating similarity to ground truth.
• Provide insight on the model's performance.

Levenshtein Similarity Ratio

• Measures similarity based on Levenshtein Distance (minimum edits needed to change one string to another).
• Simple Ratio: ((|a|+|b|) - Lev.dist(a,b)) / (|a| + |b|)
  • where |a| and |b| are the lengths of sequences a and b.
  • Lev.ratio(a, b) is the ratio.

Semantic Similarity Metrics

• BERTScore, MoverScore, and Sentence Mover Similarity rely on contextualized embeddings.
• Relatively fast and inexpensive, but sometimes have low correlation with human evaluations, lack interpretability, and possible biases.
• Cosine similarity measures the angle between embedding vectors (A and B): cosine similarity = (A · B) / (||A|| ||B||).
• Values range from -1 to 1, with 1 indicating identical vectors and -1 indicating dissimilar vectors.

Reference-Free Metrics

• Reference-free metrics do not need a reference text.
• Evaluation is based on context or source document.
• Often newer and focus on scalability as models improve.
• Includes quality-based, entailment-based, factuality-based, QA, and QG-based metrics.
• Some correlations with human evaluations are better than for reference-based, but they have drawbacks (bias).

LLM-Based Evaluators

• LLMs can be used to evaluate other LLMs.
• Offer scalable and potentially explainable evaluation.

Prompt-Based Evaluators

• LLMs evaluate text based on various criteria:
  • Text alone (fluency, coherence)
  • Generated/Original/Topic/Question (consistency, relevance)
  • Comparison to ground truth (quality, similarity)
• Various frameworks exist (Reason-then-Score, MCQ, head-to-head scoring, G-Eval, GEMBA).

LLM Embedding-Based Metrics

• Embedding models (e.g., GPT-3's text-embedding-ada-002) can calculate semantic similarity.

Metrics for LLM-Generated Code

Functional Correctness

• Evaluates accuracy of NL-to-code generation.
• Determines if generated code produces the expected output given input.
• Requires test cases and output comparison.
• Limitations: Cost of execution environments, difficulty defining comprehensive test cases, and overlooking important code aspects (e.g., style, efficiency).

Rule-Based Metrics

• Custom rules create domain-specific evaluation metrics for various tasks.

Metrics for RAG Pattern

• Retrieval-Augmented Generation (RAG) uses retrieved info from a knowledge base.
  • Generation-related metrics evaluate the generated text.
  • Retrieval-related metrics evaluate information retrieval quality from the knowledge base.