Replies to post #114 on ETF Trading

[C C]

How Internet Search Engines Work


http://computer.howstuffworks.com/internet/basics/search-engine1.htm

HOW SEARCH ENGINES WORK
AND A WEB CRAWLER APPLICATION

http://www.micsymposium.org/mics_2005/papers/paper89.pdf


cc

[C C]

Scrapy....web crawler-scraping site is a fast open-source high-level screen scraping and web crawling framework written in Python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

https://stackoverflow.com/tags/scrapy/info

difference between web-crawling and web-scraping?
list of crawlers and scraping

https://stackoverflow.com/questions/4327392/what-is-the-difference-between-web-crawling-and-web-scraping
Scrapy Tutorial

http://scrapy.readthedocs.io/en/latest/intro/tutorial.html

Web scraping, to use a minimal definition, is the process of processing a web document and extracting information out of it. You can do web scraping without doing web crawling.

Web crawling, to use a minimal definition, is the process of iteratively finding and fetching web links starting from a list of seed URL's. Strictly speaking, to do web crawling, you have to do some degree of web scraping (to extract the URL's.)

difference between these two. One refers to visiting a site, the other to extracting

https://stackoverflow.com/questions/4327392/what-is-the-difference-between-web-crawling-and-web-scraping

what’s the difference between scraping and crawling?
https://www.promptcloud.com/data-scraping-vs-data-crawling/

cc