This page is a translated version of the page Manual:Robots.txt and the translation is 62% complete.
Other languages:
Deutsch • ‎English • ‎Tiếng Việt • ‎dansk • ‎español • ‎français • ‎polski • ‎português do Brasil • ‎русский • ‎العربية • ‎فارسی • ‎ไทย • ‎中文 • ‎日本語





User-agent: *
Disallow: /






User-agent: *
Disallow: /w/


Disallow: /w


请注意,此解决方案还会导致CSS、JavaScript和图像文件被阻止,因此Google之类的搜索引擎将无法呈现Wiki文章的预览。 要解决此问题,不需要禁止整个/w目录,只需要禁止index.php

User-agent: *
Disallow: /w/index.php?

这是可行的,因为CSS和JavaScript是通过/w/load.php检索的。 另外,您也可以像在维基媒体网站矩阵上那样进行操作:

User-agent: *
Allow: /w/load.php?
Disallow: /w/


如果您没有使用短网址 ,则限制爬虫会更加困难。如果您正在运行PHP CGI并且没有美化URL,那么可以通过/index.php?title=Some_title访问文章:

User-agent: *
Disallow: /index.php?diff=
Disallow: /index.php?oldid=
Disallow: /index.php?title=Help
Disallow: /index.php?title=Image
Disallow: /index.php?title=MediaWiki
Disallow: /index.php?title=Special:
Disallow: /index.php?title=Template
Disallow: /skins/

If you are running PHP as an Apache module and you have not beautified URLs, so that articles are accessible through /index.php/Some_title:

User-agent: *
Disallow: /index.php?
Disallow: /index.php/Help
Disallow: /index.php/MediaWiki
Disallow: /index.php/Special:
Disallow: /index.php/Template
Disallow: /skins/

The lines without the colons (:) at the end restrict those namespaces' talk pages.


You may wish to omit the /skins/ restriction, as this will prevent images belonging to the skin from being accessed. Search engines which render preview images, such as Google, will show articles with missing images if they cannot access the /skins/ directory.


Disallow: /*&

because some robots like Googlebot accept this wildcard extension to the robots.txt standard, which stops most of what we don't want robots sifting through, just like the /w/ solution above. This does however, suffer from the same limitations in that it blocks access to CSS, preventing search engines from correctly rendering preview images. It may be possible to solve this by adding another line Allow: /load.php however at the time of writing this is untested.

Allow indexing of raw pages by the Internet Archiver

You may wish to allow the Internet Archiver to index raw pages so that the raw wikitext of pages will be on permanent record. This way, it will be easier, in the event the wiki goes down, for people to put the content on another wiki. You would use:

# Allow the Internet Archiver to index action=raw and thereby store the raw wikitext of pages
User-agent: ia_archiver
Allow: /*&action=raw



You can only specify what paths a bot is allowed to spider. Even allowing just the plain page area can be a huge burden when two or three pages per second are being requested by one spider over two hundred thousand pages.

Some bots have a custom specification for this; Inktomi responds to a "Crawl-delay" line which can specify the minimum delay in seconds between hits. (Their default is 15 seconds.)


Sometimes a custom-written bot isn't very smart or is outright malicious and doesn't obey robots.txt at all (or obeys the path restrictions but spiders very fast, bogging down the site). It may be necessary to block specific user-agent strings or individual IPs of offenders.

More generally, request throttling can stop such bots without requiring your repeated intervention.

An alternative or complementary strategy is to deploy a spider trap.


While robots.txt stops (non-evil) bots from downloading the URL, it does not stop them from indexing it. This means that they might still show up in the results of Google and other search engines, as long as there are external links pointing to them. (What's worse, since the bots do not download such pages, noindex meta tags placed in them will have no effect.) For single wiki pages, the __NOINDEX__ magic word might be a more reliable option for keeping them out of search results.

Warning: Display title "Manual:robots.txt/zh" overrides earlier display title "手册:Robots.txt".