For whatever reason, there are automated 'bots that read every page on a website for archiving/retrieval or other purposes.