Information Retrieval c5-c8

A method for representing documents and queries as vectors in a high-dimensional space, where each dimension corresponds to a term in the vocabulary.

The number of times a term appears in a document.

A weight assigned to a term based on how rare it is across the entire collection of documents.

A combined weight of a term, calculated as the product of term frequency (tf) and inverse document frequency (idf).

The number of documents containing a specific term.

A metric used to measure the similarity between two vectors by calculating the cosine of the angle between them.

Total count of a term's appearance across all documents in the corpus.

A technique used to handle terms that don't exist in a given document by adding a small constant (e.g., 1) to the term frequency. Prevents division by zero.

A method to decrease the size of an index by 75% in order to increase processing speed and save memory space. This is essential for efficient data handling, particularly on devices with limited storage, such as phones.

The speed at which data is moved from storage (disk) to memory (RAM), influencing the overall processing speed significantly.

The total number of documents in a dataset.

A natural language phenomenon where the frequency of a word is inversely proportional to its rank in a frequency table. The most common word is twice as frequent as the second most common, and so on.

The dictionary size in an index continually increases as the number of documents grows; there is no upper limit on the size, unlike in a normal dictionary.

The process of finding relevant documents or information based on user needs.

The degree to which a user's needs or expectations are met by a system. For example, in a search engine, getting the desired results.

Assessing if a system works as intended and if it meets user needs.

The degree to which a document satisfies a user's information need.

A research method where users perform specific tasks to evaluate a system's effectiveness.

Measuring how well search results match user needs. It requires a well-defined set of documents, queries, and relevance judgments.

A pre-selected set of documents used to evaluate search engine performance.

A judgment on whether a document is relevant to a query.

The proportion of retrieved documents that are relevant.

The proportion of relevant documents that were retrieved.

Measuring the consistency between human judges assessing relevance of documents.

A statistical measure of agreement between two raters.

A harmonic mean of precision and recall; balances both.

The percentage of correctly classified documents (relevant or not relevant).

Precision is generally prioritized in web search.

The number of times a document is cited by other documents, indicating its influence and importance.

When two documents are frequently cited together, suggesting a strong connection or shared topic.

Documents that both link to the same source document.

A method for ranking web pages based on the number and importance of incoming links, making it a key algorithm in Google's search engine.

An algorithm that assigns two scores to each page: 'authority' (number of incoming links) and 'hub' (number of outgoing links), identifying important hubs and authorities.

Using accuracy to evaluate ranked search results can be misleading, especially when considering a scenario with zero results. Zero results would be 100% accurate but doesn't reflect the actual effectiveness of the search.

Recall measures how many relevant documents are found, while precision measures how many retrieved documents are actually relevant. More documents retrieved generally increase recall but decrease precision.

A way to measure the performance of ranked search results by considering the highest precision achieved at each recall level. It represents a smoother curve of precision across different recall levels.

A single value summarizing the average precision across different queries. It considers the order of retrieved documents and is a common metric for evaluating ranked search results.

Different search queries can lead to different search results. Analyzing the variance between these queries helps understand the consistency of the search system.

Ranking search results based solely on term frequency can lead to inaccurate results, as spammy pages or pages with high frequency of a term but low relevance can appear higher.

Hyperlinks between pages can be used as an indication of quality. A page with many incoming links from relevant pages suggests a higher quality.

The text used for a hyperlink. Analyzing the text used in links can provide valuable information about the target page's content.

A method of improving search results by leveraging user input to connect anchor text with relevant pages. This helps search engines understand the relationships between terms and pages.