Help:Controlling search engine indexing

There are a variety of ways in which MediaWiki allows users to attempt to control search engine indexing, commonly termed "noindexing", as the default position is that pages are indexed. Most of the methods rely on using the noindex HTML meta tag, which tells search engines not to index certain pages. Respecting the tag, or the 1>Special:MyLanguage/Manual:robots.txt|robots.txt file, especially in terms of removing already indexed content, is up to the individual search engine, and in theory they may be ignored entirely.

The control methods are:


 * 1) Controlling an entire 1>Special:MyLanguage/namespace|namespace, via MediaWiki software settings
 * 2) Controlling classes of pages, via MediaWiki:Robots.txt
 * 3) Controlling individual pages by adding the  __NOINDEX__  1>Special:MyLanguage/Help:magic word|magic word into them, either directly or using a template made to contain it.
 * 4) Controlling pages by adding the  __NOINDEX__  magic word into standard templates used in certain situations.

Software settings
Entire namespaces can be noindexed via 1>Special:MyLanguage/Manual:$wgNamespaceRobotPolicies|$wgNamespaceRobotPolicies.

At the same time,  __NOINDEX__ </tt></> can be disabled in certain namespaces. This is controlled by the MediaWiki software setting 1>Special:MyLanguage/Manual:$wgExemptFromUserRobotsControl</>|$wgExemptFromUserRobotsControl, which defaults to 1>Special:MyLanguage/Manual:$wgContentNamespaces</>|$wgContentNamespaces.

1>Special:MyLanguage/Manual:$wgArticleRobotPolicies</>|$wgArticleRobotPolicies can be used to specify different policies for individual pages.

Robots.txt noindexing
1>Special:MyLanguage/Robots.txt</>|Robots.txt can be used to noindex pages. Unlike the software settings, it can be edited via MediaWiki, at <tvar|1>MediaWiki:Robots.txt</>.

Individual pages
Individual pages can be noindexed by adding the <tvar|code> __NOINDEX__ </tt></> 1>Special:MyLanguage/Help:Magic words</>|magic word into that page. This can be done either directly or using a template designed to contain the magic word (eg <tvar|1>NOINDEX</>); this can make use of the keyword easier to track.

Pages with the keyword are tracked by MediaWiki, and automatically categorized in a tracking category. The category name is determined by <tvar|1>MediaWiki:Noindex-category</>.

Standard template noindexing
Standard templates can include the <tvar|code> __NOINDEX__ </tt></> keyword, thereby noindexing pages to which the templates are applied. A template used to flag a particularly sensitive type of page can then automatically noindex it.

Individual pages
Individual pages can override namespace noindexing by adding the <tvar|code> __INDEX__ </tt></> 1>Special:MyLanguage/Help:magic word</>|magic word into that page. Pages with the keyword are tracked by MediaWiki, and automatically categorized in a tracking category. The category name is determined by <tvar|1>MediaWiki:Index-category</>. However, INDEX does not override noindexing via <tvar|1>MediaWiki:Robots.txt</>. It does override 1>Special:MyLanguage/Manual:$wgArticleRobotPolicies</>|$wgArticleRobotPolicies.

Current issues

 * <tvar|1></> – "Create an <tvar|code> __NOINDEX__ </tt></> equivalent to prevent indexing by internal search engine"