InvestorsHub Logo
Post# of 137
Next 10

C C

Followers 228
Posts 23772
Boards Moderated 5
Alias Born 03/30/2006

C C

Re: C C post# 113

Thursday, 07/13/2017 6:54:30 AM

Thursday, July 13, 2017 6:54:30 AM

Post# of 137
Web Crawlwers....Data minning

GOOGLE bots.....must read
https://varvy.com/googlebot.html
https://support.google.com/webmasters/answer/182072

https://sourceforge.net/projects/goldbug/files/pandamonium-webcrawler/

********** list of BOTS *****************

http://bigdata-madesimple.com/top-50-open-source-web-crawlers-for-data-mining/

https://www.elastic.co/products

http://stormcrawler.net/

***** https://www.nohodo.com/ *****

**** http://sp.nohodo.com/signup/ ****

******** Robots Database ********

read about Robots usage rules.....https://moz.com/learn/seo/robotstxt

http://www.robotstxt.org/

http://www.robotstxt.org/db.html

ABCdatos BotLink
Acme.Spider
Ahoy! The Homepage Finder
Alkaline
Anthill
Walhello appie
Arachnophilia
Arale
Araneo
AraybOt
ArchitextSpider
Aretha
ARIADNE
arks
AskJeeves
ASpider (Associative Spider)
ATN Worldwide
Atomz.com Search Robot
AURESYS
BackRub
Bay Spider
Big Brother
Bjaaland
BlackWidow
Die Blinde Kuh
Bloodhound
Borg-Bot
BoxSeaBot
bright.net caching robot
BSpider
CACTVS Chemistry Spider
Calif
Cassandra
Digimarc Marcspider/CGI
Checkbot
ChristCrawler.com
churl
cIeNcIaFiCcIoN.nEt
CMC/0.01
Collective
Combine System
Conceptbot
ConfuzzledBot
CoolBot
Web Core / Roots
XYLEME Robot
Internet Cruiser Robot
Cusco
CyberSpyder Link Test
CydralSpider
Desert Realm Spider
DeWeb(c) Katalog/Index
DienstSpider
Digger
Digital Integrity Robot
Direct Hit Grabber
DNAbot
DownLoad Express
DragonBot
DWCP (Dridus' Web Cataloging Project)
e-collector
EbiNess
EIT Link Verifier Robot
ELFINBOT
Emacs-w3 Search Engine
ananzi
esculapio
Esther
Evliya Celebi
FastCrawler
Fluid Dynamics Search Engine robot
Felix IDE
Wild Ferret Web Hopper #1, #2, #3
FetchRover
fido
Hämähäkki
KIT-Fireball
Fish search
Fouineur
Robot Francoroute
Freecrawl
FunnelWeb
gammaSpider, FocusedCrawler
gazz
GCreep
GetBot
GetURL
Golem







cc



which ever way the HERD goes....GO the other way

Join the InvestorsHub Community

Register for free to join our community of investors and share your ideas. You will also get access to streaming quotes, interactive charts, trades, portfolio, live options flow and more tools.