Web Crawler

• Web crawler is a program that accesses a web site and traverses through the site by following the links present on the pages.
• Whenever we search for a keyword and go to a site.
• Google engine collects information on the site clicked by the user using its crawler tool.
• Googlebot is Google’s web crawling tool, which finds and retrieves pages on the web and hands them off to the Google indexer.

Good Reads
Crawling a web sites with HtmlAgilityPack

Experiences with the WebCrawler Anatomy of a Search Engine 
How Google Works
Web crawler architectures
Google vs. Bing: Correlation Analysis of Ranking Elements

Bing vs. Google: Prominence of Ranking Elements
How To: Detect Web Crawlers, Spiders, & Robots in ASP.NET
How to detect search engine crawlers?
Web Crawler
How Search Engines Work
Distributed web crawling
History of Search: Search Engine Timeline