"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;

September 05, 2010

Web Crawler

  • Web crawler is a program that accesses a web site and traverses through the site by following the links present on the pages.
  • Whenever we search for a keyword and go to a site.
  • Google engine collects information on the site clicked by the user using its crawler tool.
  • Googlebot is Google’s web crawling tool, which finds and retrieves pages on the web and hands them off to the Google indexer.

Good Reads
Crawling a web sites with HtmlAgilityPack
Bing vs. Google: Prominence of Ranking Elements
How To: Detect Web Crawlers, Spiders, & Robots in ASP.NET
How to detect search engine crawlers?
Web Crawler
How Search Engines Work
Distributed web crawling
History of Search: Search Engine Timeline

No comments: