Crawlers can retrieve data much quicker and in greater depth than human searchers, so they can have a crippling impact on the performance of a site. Needless to say if a single crawler is performing multiple requests per second and/or downloading large files, a server would have a hard time keeping up with requests from multiple crawlers.
Source: http://en.wikipedia.org/wiki/Web_crawler
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment