Search Engine Indexing and Crawling

Search Engines

Indexing

• Process of creating a massive database of web pages, known as an index
• Index is used to quickly retrieve and rank web pages in response to user queries
• Indexing involves:
  • Tokenization: breaking down web pages into individual words or tokens
  • Stopword removal: removing common words like "the", "and", etc. that don't add value
  • Stemming or Lemmatization: reducing words to their base form (e.g., "running" becomes "run")

Crawling

• Process of automatically discovering and fetching web pages to be indexed
• Crawling involves:
  • Web crawlers or spiders: software programs that continuously scan the web for new pages
  • Seed URLs: initial URLs used to start the crawling process
  • Link extraction: identifying and following hyperlinks to discover new pages
  • Page prioritization: determining which pages to crawl first based on factors like importance and freshness

Relevance Ranking

• Process of ranking web pages in response to a user query based on their relevance
• Relevance ranking involves:
  • Query parsing: breaking down the user query into individual keywords and phrases
  • Document scoring: calculating a score for each web page based on factors like:
    • Term frequency: how often the keywords appear on the page
    • Inverse document frequency: how rare the keywords are across the entire index
    • Link analysis: the importance of the page based on its inbound and outbound links
  • Result ranking: sorting the scored web pages to display the most relevant results to the user

Search Engines

Indexing

• A massive database of web pages is created, known as an index, to facilitate quick retrieval and ranking of web pages in response to user queries.
• Indexing involves tokenization, which breaks down web pages into individual words or tokens.
• Stopword removal is also part of the indexing process, where common words like "the" and "and" are removed as they don't add value.
• Stemming or Lemmatization is used to reduce words to their base form, such as "running" becoming "run".

Crawling

• Web crawlers or spiders continuously scan the web for new pages, using seed URLs as a starting point.
• Crawling involves link extraction, where hyperlinks are identified and followed to discover new pages.
• Page prioritization is used to determine which pages to crawl first, based on factors like importance and freshness.

Relevance Ranking

• Query parsing breaks down the user query into individual keywords and phrases.
• Document scoring calculates a score for each web page based on term frequency, inverse document frequency, and link analysis.
• Term frequency refers to how often the keywords appear on the page.
• Inverse document frequency refers to how rare the keywords are across the entire index.
• Link analysis determines the importance of the page based on its inbound and outbound links.
• Result ranking sorts the scored web pages to display the most relevant results to the user.