Untitled Quiz

Evaluation Methods

• Evaluation involves comparing performance against a benchmark, whether it's a mathematical model, user expectations, or some other criterion.
• Deciding what constitutes "good" performance is a key initial step.
• Performance can be evaluated through comparison with a mathematical model, or the expectations of users.

User-Based Evaluation

• User biases can affect evaluation results. Analyst expectations can cause misinterpretations of feature performance.

• Even metrics for thought processes can't completely eliminate bias.

• User input is sometimes needed for high-level tests, with a statistical model needed to underpin these tests.

• Blind analyses (where tests are designed, run, and assessed independently) are preferable

• Design tests using background or simulation data.

• Avoid adjusting analyses when looking at the final dataset.

• Movie reviews often exhibit bias. Users tend to rate movies they like higher than average. This bias needs correction.

• Search engine users have different expectations. Different types of search results are needed. For example, some users may want academic articles, while others may want magazine articles.

• PageRank and quality aren't enough to evaluate search engines effectively.

• The addition of recommendations systems can be needed to refine search engine results.

Model-Based Evaluation

• A measurement is compared against a model.
• A model provides a central value and an uncertainty band.
• To proceed, an observable is chosen, and the expected value is determined.

Evaluation of Search Systems

• Search system performance needs to be quantified. Key questions to be asked include;
• Is the correct information being retrieved?
• Is the information being returned in the correct order?
• Are results being returned in a timely manner?
• Are results displayed appropriately?
• Evaluation methods need to be defined to answer these questions.
• Potential uncertainties of the evaluation method must be taken into account.

Recall

• Recall measures the proportion of relevant documents retrieved from the total set of relevant documents.
• A higher recall value means a larger proportion of relevant documents were retrieved.
• Formula: Recall = (Relevant retrieved documents ∩ Retrieved documents)/Total relevant documents

Precision

• Precision measures the proportion of retrieved documents that are relevant to the total number of retrieved documents.
• A higher precision value implies that a higher fraction of retrieved documents are relevant.
• Formula: Precision = (Relevant retrieved documents ∩ Retrieved documents)/Total Retrieved documents

Recall and Precision - Example Data

• Example data from various queries shows how recall and precision can vary. Data suggests that precision is typically higher for the first documents retrieved in a search.

Evaluation of Partial Sets

• Rank order can be used to evaluate queries.
• Properties of earlier results are often more important.
• Precision@k measures the proportion of top-k results that are relevant.
• Recall@k measures the proportion of relevant documents that are among the top-k results.

Average Precision (AP) and Mean Average Precision (MAP)

• Average Precision (AP) is the average precision for a given threshold or rank of documents in a search query.
• Mean Average Precision (MAP) is the average of the average precision for multiple queries.
• Individual words retrieved can be weighted to be more impactful that those retrieved later in the order.
• The precision is typically higher for the first rank

Other Evaluation Metrics

• Other metrics include "Bpref"; (relevant documents ranked higher than irrelevant documents);
• Relative recall and
• Geometric Mean Average Precision (GMAP).
• Statistical tests can determine if one system is better than another.

Test Data

• Test data needs to include tagged relevance.
• Individual queries need to be tagged as relevant.
• Average performance for each system can then be calculated.

Information Retrieval Evaluation

• Early evaluation primarily used Cranfield tests on library systems.
• More recent studies use simulations built from Text REtrieval Conference (TREC) data.

System Evaluation: Test Collection

• A wide range of document types (text, images, videos, speech, etc), are selected.
• Collection creators often use user-provided information requests

CISI and WSJ Collections

• The CISI collection contains 1,430 documents and 112 queries, with an average of 41 relevant documents per query.
• The WSJ collection contains 74,520 documents (from Wall Street Journal newspaper articles) and 50 topics; approximately.

TREC

• TREC provides many different datasets and tools for interface with test data.

Pooling

• Pooling is a technique used to evaluate large collections when manual assessment is needed.
• Random sampling from an assessed document pool is used
• Often better to include several systems to initially generate the pool

Web as Test Collection

• Evaluating web search involves a massive amount of data (billions of pages) and short, dynamic query terms. Creating a snapshot of this data is required, then the pooling method can be applied.

Evaluation of Recommender Systems

• Static data with user ratings is used

Training and Testing

• Data is split into training and testing sets. Training sets are used to train the recommender system, and testing sets are used to evaluate the trained system

K-Fold Cross-Validation

• Bias is reduced by repeating training and testing from different sections of the dataset.

N-1 Testing

• This method evaluates a recommender system for a single active user, where data for one user is deliberately withheld from training.

Confusion Matrix

• A confusion matrix is used to evaluate the effectiveness of a classification system. It shows outcomes and predictions of an evaluation.
• Common confusion matrix measures include precision; recall; accuracy and; F1 (harmonic mean).

Prediction Accuracy

• Error values are defined to measure accuracy. This data is used to train machine learning algorithms.
• Error formulae include Root-Mean-Square Error (RMSE) and Mean Absolute Error (MAE).

Significance Testing

• Statistical testing compares the effectiveness of systems (A vs B).
• The effect, if statistically significant, can be evaluated with likelihood functions, and systematic errors taken into account.

A/B Testing

• A/B testing is a controlled experiment with two options or versions that diverts users to test an environment.
• Used to evaluate the impact of new features, or compare differences.

Other Evaluation Approaches

• Prototypical users can be involved in test tasks to assess user satisfaction.
• Previous system usage can also be reviewed with an analysis of accumulated logs to improve overall understanding of system performance.