The objective of this article is to provide a general overview of how search robots (also called spiders) go about crawling and indexing web pages.

• 目的这篇文章的意思是提供一个概述如何搜索机器人(也叫蜘蛛)到约爬行和索引网页。