IR c9-c10

Study Notes

PageRank and HITS are algorithms used to determine the importance of web pages.
HITS (Hubs and Authorities) assigns two values to each page: hub value and authority value.
Hub value represents the number of links pointing from a page to other pages.
Authority value represents the number of links pointing to a page from other pages.
A high authority score indicates a page with quality information that other pages link to.
A high hub value indicates a page that links to many important pages.

PageRank is a widely used algorithm for determining web page importance.
It measures the importance of a page based on the importance of the pages linking to it.
The algorithm considers all pages as a graph with links representing edges.
Each page in the graph receives an initial PageRank score.
Each page's PageRank score is calculated iteratively.
The PageRank score of a page is a function of the PageRank scores of all the pages linking to it.
The calculation accounts for the number of outgoing links from each page.
The algorithm converges after a certain number of iterations, providing a normalized PageRank value for each page.

The base set is the set of pages relevant to a given query.
The base set comprises pages directly related to a query and those linked to the directly related ones.
Authority score (a(i)) of a page is calculated as the sum of hub scores (h(j)) of all pages that link to it.
Hub score (h(i)) of a page is calculated as the sum of authority scores (a(j)) of all pages that it links to.

Small-world networks exhibit a high clustering coefficient and a short average path length between any two nodes.
The clustering coefficient measures the density of connections among neighbors of a node.
The average path length measures the average number of steps needed to reach any node from any other node.
Such networks show characteristics of both regular networks and random networks.
Real-world examples of small-world networks include social networks and biological networks.

Web crawlers are automated programs that traverse the web by following links between pages.
Crawlers collect information, store it, and index it for search engines and other users to find relevant content.
One of the steps in crawling is URL initialization and the identification of initial set of URLs.
Crawlers follow links within pages to follow a set of path and collect data from web pages.
The crawling process must be well-organized and manageable
Politeness, freshness and robustness are important factors for web crawlers.

URL Frontier is a system for managing and prioritizing URLs to be crawled.
Queues are used to organize URLs to ensure appropriate crawling.
Queues also help to maintain a priority between various pages to be crawled.
It also ensures that the politeness and freshness of the crawled pages are maintained.