Very important for internet marketing / search engine optimization (SEO).
Web crawlers, or spiders, are software applications that automatically visit (crawl) pages on the Internet with the primary purpose indexing the pages and gathering selected information on the site. The task performed by the crawlers is very fast and this allows search engines, for example, to monitor and index information on the Web. The task would not be possible without an automated process.
More simply, a crawler is a search engine program that “crawls” the web, collecting data, following links, making copies of new and updated sites, and storing URLs in the search engine’s Index.