Help:Controlling search engine indexing

Note: When you edit this page, you agree to release your contribution under the CC0. See Public Domain Help Pages for more info. Some old revisions for this page were imported under CC BY-SA license. Only new contributions are PD.

There are a variety of ways in which MediaWiki allows users to attempt to control search engine indexing, commonly termed "noindexing", as the default position is that pages are indexed. Most of the methods rely on using the noindex HTML meta tag, which tells search engines not to index certain pages. Respecting the tag or the robots.txt file, especially in terms of removing already indexed content, is up to the individual search engine, and in theory they may be ignored entirely.

The control methods are:

1. Controlling an entire namespace, via MediaWiki software settings
2. Controlling classes of pages, via MediaWiki:Robots.txt
3. Controlling individual pages by adding the __NOINDEX__ magic word into them, either directly or using a template made to contain it.
4. Controlling pages by adding the __NOINDEX__ magic word into standard templates used in certain situations.

Indexing policy set by one of the means above only apply for the page content only. Views that are a part of MediaWiki interface (such as the page history, historical page version or most special pages) are never indexed, irrespective of the settings set above.

Software settings and robots.txt

Software settings

Entire namespaces can be noindexed via $wgNamespaceRobotPolicies .

At the same time, __NOINDEX__ can be disabled in certain namespaces. This is controlled by the MediaWiki software setting $wgExemptFromUserRobotsControl , which defaults to $wgContentNamespaces .

$wgArticleRobotPolicies can be used to specify different policies for individual pages.

Robots.txt noindexing

System message MediaWiki:Robots.txt can be used to noindex pages. Unlike the software settings, it can be directly edited.

NOINDEX magic word

Individual pages

Individual pages can be noindexed by adding the __NOINDEX__ magic word into that page. This can be done either directly or using a template designed to contain the magic word (eg {{NOINDEX }}); this can make use of the keyword easier to track.

Pages with the keyword are tracked by MediaWiki, and automatically categorized in a tracking category. The category name is determined by MediaWiki:Noindex-category.

Standard template noindexing

Standard templates can include the __NOINDEX__ keyword, thereby noindexing pages to which the templates are applied. A template used to flag a particularly sensitive type of page can then automatically noindex it.

INDEX magic word

Individual pages

Individual pages can override namespace noindexing by adding the __INDEX__ magic word into that page. Pages with the keyword are tracked by MediaWiki, and automatically categorized in a tracking category. The category name is determined by MediaWiki:Index-category. However, INDEX does not override noindexing via MediaWiki:Robots.txt. It does override $wgArticleRobotPolicies .

Current issues

• phab:T24251 – Magic word to remove page from internal MediaWiki search results