Help:CirrusSearch

CirrusSearch is a new search engine for MediaWiki. The Wikimedia Foundation migrated to CirrusSearch since it features key improvements over the previously used search engine, LuceneSearch.

This page describes the features that are new or different compared to the past solutions.

Frequently asked questions
If your question is not answered here, feel free to ask on the talk>Help talk:CirrusSearch|talk page and someone will answer it for you.

What's improved?
The new search engine features three main improvements over the old search engine, namely:


 * Better support for searching in different languages.


 * Faster updates to the search index, meaning changes to articles are reflected in search results much faster.


 * Expanding templates, meaning that all content from a template is now reflected in search results.

Updates
Updates to the search index are done in near real time.

You should be able to search for your changes as soon as you make them.

Changes to templates should take effect in articles that include the template in a few minutes.

The templates changes use the job queue, so performance may vary.

A purge>Special:MyLanguage/Manual:Purge|null edit to the article will force the change through, but that shouldn't be required if everything is going well.

Search suggestions
The search suggestions you get when you type into the search box that drops down candidate pages is substantively the same with articles sorted by the number of incoming links.

Worth noting is that if you start your search with ~ we won't find any articles as you type and you can safely hit enter at any time to jump to the search results page.

ASCII/accents/diacritics folding is turned on for English text, but there are some formatting problems with the result. See [https://bugzilla.wikimedia.org/show_bug.cgi?id=52656 52656].

Full text search
Full text search (the kind that lands you on the search results page) searching in title, redirects, headings, and article text so it shouldn't present any surprises. The big change here is that templates are expanded.

Stemming
There is support for dozens of languages, but all languages are wanted.

There is a list of currently supported languages at [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html elasticsearch.org]; see their [http://www.elasticsearch.org/contributing-to-elasticsearch/ documentation on contributing] to submit requests or patches.

Filters (intitle:, incategory: and linksto:)


We've tightened up the syntax around these quite a bit.


 * intitle:foo
 * Find articles whose title contains foo. Stemming is enabled for foo.
 * intitle:"fine line"
 * Find articles whose title contains fine then line. Stemming is enabled. Matches The finest (lines) but not The finest ever lines.
 * intitle:foo bar
 * Find articles whose title contains foo and whose title or text contains bar.
 * -intitle:foo bar


 * Find articles whose title does not contain foo and whose title or text contains bar.
 * intitle: foo bar
 * Syntax error, devolves into searching for articles whose title or text contains intitle:, foo, and bar.
 * incategory:Music
 * Find articles that are in Category:Music
 * incategory:"music history"
 * Find articles that are in Category:Music_history


 * incategory:"musicals" incategory:"1920"
 * Find articles that are in both Category:Musicals and Category:1920
 * -incategory:"musicals" incategory:"1920"
 * Find articles that are not in Category:Musicals but are in Category:1920
 * cow*
 * Find articles whose title or text contains words that start with cow
 * linksto:Help:CirrusSearch
 * find articles that link to a page
 * -linksto:Help:CirrusSearch CirrusSearch
 * find articles that mention CirrusSearch but do not link to the page Help:CirrusSearch

prefix:
The prefix: syntax in its current form is relied upon for a great deal of functionality so it's been recreated as exactly as possible.

Note that the old rule of having to put prefix: at the end of the query still applies.

Special prefixes

 * Find articles whose text is most similar to the text of the given articles.
 * Find articles whose text is most similar to the text of the given articles.


 * The  query works by choosing a set of words in the input articles and run a query with the chosen words. You can tune the way it works by adding the following parameters to the search results URL:
 * : Minimum number of documents (per shard) that need a term for it to be considered.
 * : Maximum number of documents (per shard) that have a term for it to be considered.
 * : Maximum number of terms to be considered.
 * : Minimum number of times the term appears in the input to doc to be considered. For small fields this value should be 1.
 * : Minimal length of a term to be considered. Defaults to 0.
 * : The maximum word length above which words will be ignored. Defaults to unbounded (0).
 * (comma separated list of values): These are the fields to use. Allowed fields are,  ,  ,  ,   and.
 * ( | ): use only the field data. Defaults to : the system will extract the content of the   field to build the query.
 * : The percentage of terms to match on. Defaults to 0.3 (30 percent).
 * Example:
 * These settings can be made persistent by overriding  in Help:System message.
 * Find articles in the talk namespace whose title or text contains the word foo
 * Find articles in the talk namespace whose title or text contains the word foo


 * Find articles in the file namespace on this wiki and commons whose title or text contains the word
 * You can add  to the query (like  ) to remove the results from commons
 * You can add  to the query (like  ) to remove the results from commons
 * You can add  to the query (like  ) to remove the results from commons

Did you mean
"Did you mean" suggestions are designed to notice if you misspell an uncommon phrase that happens to be an article title. If so, they'll let you know. They also seem to suggest more things than they ought to sometimes.

Prefer phrase matches
If you don't have too much special syntax in your query we'll give perfect phrase matches a boost. I'm being intentionally vague because I'm not sure exactly what "too much special syntax" should be. Right now if you add any explicit phrases to your search we'll turn off this feature.

Fuzzy search
Putting a ~ after a search term (but not double quotes) activates fuzzy search. You can also put a number from 0 to 1 to control the "fuzziness" fraction, e.g. nigtmare~.9 or lighnin~.1 or lighnin~0.1. Closer to one is less fuzzy.

Phrase search and proximity
Surrounding some words with quotes declares that you are searching for those words close together. You can add a ~ and then a number after the second quote to control just how close you mean. The proper name for this "closeness" is "phrase slop". The default "phrase slop" is 1.

Quotes and exact matches
Quotes turn on exact term matches. You can add a ~ to the quote to go back to the more aggressive matcher you know and love.

prefer-recent:
You can give recently edited articles a boost in the search results by adding "prefer-recent:" to the beginning of your search.

By default this will scale 60% of the score exponentially with the time since the last edit, with a half life of 160 days.

This can be modified like this: "prefer-recent:,".

proportion_of_score_to_scale must be a number between 0 and 1 inclusive.

half_life_in_days must be greater than 0 but allows decimal points.

This number works pretty well if very small.

I've tested it around .0001, which is 8.64 seconds.

This will eventually be on by default for Wikinews, but there is no reason why you can't activate it in any of your searches.

hastemplate:
You can find pages that use a certain template by adding the filter  to the search. We provide for the usual "syntactic sugar" of template calls. This means the lenient pagename and fullpagename capitalization works, and the main namespace abbreviation, ":" works. For example to find which pages transclude Quality image the full search (in all your preferred namespaces) can be: , and for that same template name in the main namespace, this works. You can omit the quotes if the template title does not contain a space. will filter pages that do not contain that template.

For wikitext that calls a template directly, you can use insource:, but hastemplate: searches the "post-expansion inclusion", so hastemplate: can find a template acting only temporarily as a "secondary template" or "meta-template", which are seen in neither the source nor content, ( but only included as a helper to any other template producing the final content). All content from a template is now reflected in search results is still the relevant philosophy here.

boost-templates:
You can boost pages' scores based on what templates they contain. This can be done directly in the search via  or you can set the default for all searches via the new   message. replaces the contents of  if the former is specified. The syntax is a bit funky but was chosen for simplicity. Some examples:


 * Find files in the China category sorting quality images first.
 * Find files in the China category sorting quality images first.


 * Find files in the China category sorting quality images first and low quality images last.
 * Find files in the China category sorting quality images first and low quality images last.


 * Find files about popcorn sorting quality images first and low quality images last. Remember that through the use of the  message this can be reduced to just.
 * Find files about popcorn sorting quality images first and low quality images last. Remember that through the use of the  message this can be reduced to just.

Don't try to add decimal points to the percentages. They don't work and search scoring is such that they are unlikely to matter much.

A word of warning about : if you add really really big or small percentages they can poison the full text scoring. Think, for example, if enwiki boosted featured articles by a million percent. Then searches for terms mentioned in featured articles would find the featured articles before exact title matches of the terms. Phrase matching would be similarly blown away so a search like  would find a featured article with those words scattered throughout it instead of the article for Brave New World.

Sorry for the inconsistent  in the name. Sorry again but the quotes are required on this one. Sorry also for the funky syntax. Sorry we don't try to emulate the template transclusion syntax like we do with.

insource:
This can pick up template arguments, URLs, links, html, etc. It has two forms, one is an indexed search, and the other is regex based.

Tip: Instead of running a bare insource:/regexp/ on an entire namespace or on the wiki, these return much much faster when you limit their search domain to the results of one or more index-based searches: Regex are character-wise per-page searches. All other search terms use an index.
 * uses a wikitext phrase filter or wikitext proximity filter.
 * uses a wikitext phrase filter or wikitext proximity filter.
 * uses a wikitext phrase filter or wikitext proximity filter. All non-alphanumeric character strings were being translated to a single space anyway.
 * uses a single pagename that works in any edit box. Excellent practice for developing a new or complex regexp.
 * uses a prefix filter that includes a single username.
 * uses a prefix filter that includes multiple usernames.
 * searches an entire namespace.
 * searches an unknowable domain: the user's default search domain. (Settable at Special:Search, it is commonly reset by power users from the default, main, to ALL namespaces.)

To develop a new regexp, or refine a complex regexp, use  in any edit box, or search your own subpages:

To search for regular expression metacharacters literally, you must "escape" them, usually one at a time with a backslash. For example / which matches a literal dash, dot, or square bracket.

You can also "quote" a string (set) of characters using double-quotes to remove the metacharacter meaning: /

Inside double quotes you must use backslash-escape to escape the double-quote character, for example.

Inside the regexp you must use the backslash-escape to quote the slash character that is the closing delimiter.

The square-bracket notation (for creating your own character-class) also escapes metacharacters. You can escape any metacharacter that happens to be included, except that the right square bracket must be escaped or a dash must be backslash escaped. them as just shown, or just the first position can be used:  or , either of which match a dot, dash or right square bracket.

For the actual meaning of the metacharacters see the [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html#regexp-syntax explanation of the syntax]. For the formal definition see the [http://lucene.apache.org/core/4_8_1/core/org/apache/lucene/util/automaton/RegExp.html Lucene grammar for regular expressions].

For example, using metacharacters to find the usage of a template called Val having, inside the template call, an unnamed parameter containing a possibly signed, three to four digit number, possibly surrounded by space characters, AND on the same page, inside a template Val call, a named argument "fmt=commas" having any allowable spaces around it, (it could be the same template call, or a separate one):

It is fast because it uses two filters so that every page the regexp crawls has the highest possible potential. Assuming my search domain is set to ALL, it searches the entire wiki.

Auxiliary text
Cirrus considers some text in the page to be "auxiliary" to what the page is actually about. Examples include table contents, image captions, and "This article is about the XYZ. For ZYX see ZYX" style links. You can also mark article text as auxiliary by adding the  class to the html element containing the text.

Auxiliary text is worth less than the rest of the article text and it is in the snippet only if there are no main article snippets matching the search.

Lead text
Cirrus assumes that non-auxiliary text that is between the top of the page and the first heading is the "lead in" paragraph. Matches from the lead in paragraph are worth more in article ranking.

Commons search
By default when the search contains the file namespace, Cirrus will search commons as well.

You can disable this behavior by adding  to the search.

If you are using a namespace prefix to select the namespace the syntax looks like.

If you aren't using a namespace prefix to select the namespace then the syntax looks like.