Web Crawling Fundamentals

Basic Crawler Operation

• Crawler operation starts with known seed URLs
• Fetch and parse these URLs, extracting new URLs they point to
• Place extracted URLs on a queue for further processing

Complications

• Malicious pages can be encountered, including spam pages and spider traps
• Spider traps can cause a crawler to make an infinite number of requests or crash
• Non-malicious pages can also pose challenges, such as varying latency and bandwidth
• Webmasters may have stipulations that need to be considered

Spider Traps

• A spider trap is a set of web pages that intentionally or unintentionally cause a crawler to make an infinite number of requests or crash
• Spider traps can be dynamically generated

Robustness and Politeness

• A crawler must be robust to handle spider traps and malicious behavior
• A crawler must be polite and respect implicit and explicit politeness considerations

Politeness Considerations

• Explicit politeness: robots.txt specifies what portions of a site can be crawled
• Implicit politeness: avoid hitting a site too often, even with no explicit specification

Additional Crawler Requirements

• Be capable of distributed operation to run on multiple machines
• Be scalable to increase the crawl rate by adding more machines
• Optimize performance and efficiency to use available processing and network resources
• Fetch pages of higher quality first
• Continuously operate and fetch fresh copies of previously fetched pages
• Be extensible to adapt to new data formats and protocols

URL Frontier

• The URL frontier can include multiple pages from the same host
• Avoid trying to fetch all pages from the same host at the same time
• Keep all crawling threads busy to optimize performance