Large Robots document literacy how to use robots to improve website ranking

Robots file is a web site and spiders between "gentleman’s agreement" – Robots file not only can save the resource of the website, also can help more effectively grasp the spider net, so as to improve the ranking.

* /






For example:

said shielding all spiders. When we do the pseudo static treatment, at the same time dynamic pages and static web pages, web content as the page state as like as two peas, mirror, so we need to shield the dynamic web page can use * to shield the dynamic web pages

"Disallow:/folder/" said interception is a directory, all files in the directory under the file is not allowed to crawl, but allowed to grab folder.hlml.

"Disallow:/folder": all files under /folder/ and folder.html can be crawled.

high complex: 贵族宝贝mygaofu贵族宝贝, please indicate the link

can also open the good website, see their robots is how to write the file, and then modify according to their own needs. The Robots file can let the spider to spend more time to grab the content, so the optimization of the robots file is necessary.

3: "*" matches any character

4:$, the end of



if you want to stop all reptiles Google BOT:



Uer-agent: allows spider

2: "/folder/" and "/folder" from


if you want to intercept the end with a string of the web site, you can use the $, for example, to intercept the.Asp at the end of the address:



1: only allow Google bot

. This article from the Dongyang

Leave a Reply

Your email address will not be published. Required fields are marked *