Web Crawlwers....Data minning GOOGLE bots.....must read https://varvy.com/googlebot.html https://support.google.com/webmasters/answer/182072 https://sourceforge.net/projects/goldbug/files/pandamonium-webcrawler/ ********** list of BOTS ***************** http://bigdata-madesimple.com/top-50-open-source-web-crawlers-for-data-mining/ https://www.elastic.co/products http://stormcrawler.net/ ***** https://www.nohodo.com/ ***** **** http://sp.nohodo.com/signup/ **** ******** Robots Database ******** read about Robots usage rules.....https://moz.com/learn/seo/robotstxt http://www.robotstxt.org/ http://www.robotstxt.org/db.html ABCdatos BotLink Acme.Spider Ahoy! The Homepage Finder Alkaline Anthill Walhello appie Arachnophilia Arale Araneo AraybOt ArchitextSpider Aretha ARIADNE arks AskJeeves ASpider (Associative Spider) ATN Worldwide Atomz.com Search Robot AURESYS BackRub Bay Spider Big Brother Bjaaland BlackWidow Die Blinde Kuh Bloodhound Borg-Bot BoxSeaBot bright.net caching robot BSpider CACTVS Chemistry Spider Calif Cassandra Digimarc Marcspider/CGI Checkbot ChristCrawler.com churl cIeNcIaFiCcIoN.nEt CMC/0.01 Collective Combine System Conceptbot ConfuzzledBot CoolBot Web Core / Roots XYLEME Robot Internet Cruiser Robot Cusco CyberSpyder Link Test CydralSpider Desert Realm Spider DeWeb(c) Katalog/Index DienstSpider Digger Digital Integrity Robot Direct Hit Grabber DNAbot DownLoad Express DragonBot DWCP (Dridus' Web Cataloging Project) e-collector EbiNess EIT Link Verifier Robot ELFINBOT Emacs-w3 Search Engine ananzi esculapio Esther Evliya Celebi FastCrawler Fluid Dynamics Search Engine robot Felix IDE Wild Ferret Web Hopper #1, #2, #3 FetchRover fido Hämähäkki KIT-Fireball Fish search Fouineur Robot Francoroute Freecrawl FunnelWeb gammaSpider, FocusedCrawler gazz GCreep GetBot GetURL Golem cc