Manual:Robot policy
Robot policy is used to tell web crawlers and other tools what to exclude.
Configuration settings
edit- $wgDefaultRobotPolicy - Default robot exclusion policy for the wiki.
- $wgArticleRobotPolicies - Allow customization of robot policies on a per-article basis.
- $wgExemptFromUserRobotsControl - An array of namespace keys in which the __INDEX__/__NOINDEX__ magic words will not function.
- $wgNamespaceRobotPolicies - Allow customization of robot policies on a per-namespace basis.