Manual:Robot policy

Robot policy is used to tell web crawlers and other tools what to exclude.

Configuration settings
Allow customization of robot policies on a per-article basis. Default robot exclusion policy for the wiki. An array of namespace keys in which the  __INDEX__/__NOINDEX__  magic words will not function. Allow customization of robot policies on a per-namespace basis.