Manual:$wgDefaultRobotPolicy/en

Details
Allows specifying the default robot policy for all pages on the wiki. The default is to encourage indexing and following of links. This can be overridden on a per-namespace and/or per-article basis.

e.g.

would forbid indexing and following of links on all pages outside the main namespace.

External link

 * About the Robots  tag (www.robotstxt.org)