Very important for internet marketing / search engine optimization (SEO).

Web crawlers, or spiders, are software applications that automatically visit (crawl) pages on the Internet with the primary purpose indexing the pages and gathering selected information on the site. The task performed by the crawlers is very fast and this allows search engines, for example, to monitor and index information on the Web. The task would not be possible without an automated process.