Help:CirrusSearch

CirrusSearch is a MediaWiki extension that uses Elasticsearch to provide enhanced search features over the default MediaWiki search. The Wikimedia Foundation uses CirrusSearch for all Wikimedia projects.

This page describes the features of CirrusSearch.

If your question is not answered here, feel free to ask on the talk>Help talk:CirrusSearch|talk page and someone will answer it for you.

For information on the MediaWiki extension see Extension:CirrusSearch.

What's improved?
CirrusSearch features three main improvements over the default MediaWiki search, namely:


 * Better support for searching in different languages.


 * Faster updates to the search index, meaning changes to articles are reflected in search results much faster.


 * Expanding templates, meaning that all content from a template is now reflected in search results.

How frequently is the search index updated?
Updates to the search index are done in near real time.

Changes to pages should appear immediately in the search results.

Changes to templates should take effect in articles that include the template in a few minutes.

The templates changes use the job queue, so performance may vary.

A purge>Special:MyLanguage/Manual:Purge|null edit to the article will force the change through, but that shouldn't be required if everything is going well.

Search suggestions
The search suggestions you get when you type into the search box that drops down candidate pages is substantively the same with articles sorted by the number of incoming links.

Search suggestions can be skipped and queries will go directly to the search results page. Add a tilde "~" before the query. Example "~Frida Kahlo". The search suggestions will still appear, but hitting the Enter key at any time will take you tot the search results page.

ASCII/accents/diacritics folding is turned on for English text, but there are some formatting problems with the result. See.

Full text search
A "full text search" is an "indexed search". All pages are stored in the wiki database, and all the words in them are stored in the search database, which is an index to the full text of the wiki. Each visible word is indexed to the list of pages where it is found, so a search for a word is as fast as looking up a single-record. Furthermore, for any changes in wording, the search index is updated within seconds.

There are many indexes on the "full text" of the wiki to facilitate the many types of searches needed. The full wikitext is indexed many times into many special-purpose indexes, each parsing the wikitext in whatever way optimizes their use. Example indexes include
 * "auxiliary" text, includes hatnotes, captions, ToC, and any wikitext classed by an HTML attribute class=searchaux.
 * "Lead-in" text is the wikitext between the top of the page and the first heading.
 * The "category" text indexes the listings at the bottom.
 * Templates are indexed. If the transcluded words of a template change, then all the pages that transclude it are updated. (This can take a long while depending on a job queue.) If the subtemplates used by a template change, the index is updated.
 * Document content that is stored in the File/Media namespace are now indexed. Thousands of formats are recognized.

There is support for dozens of languages, but all languages are wanted.

There is a list of currently supported languages at [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html elasticsearch.org]; see their [http://www.elasticsearch.org/contributing-to-elasticsearch/ documentation on contributing] to submit requests or patches.

CirrusSearch will optimize your query, and run it. The resulting titles are weighted by relevance, and heavily post-processed, 20 at a time, for the search results page. For example snippets are garnered from the article, and search terms are highlighted in bold text.

Search results will often be accompanied by various preliminary reports including Did you mean (spelling correction), and, when no results would otherwise be found it will say Showing results for (query correction) search instead for (your query).

Search features also include
 * sorting navigation suggestions by the number of incoming links.
 * Starting with the tilde character ~ to disable navigation and suggestions in such a way that also preserves page ranking.
 * Smart-matching characters by normalizing (or "folding") non-keyboard characters into keyboard characters.
 * Words and phrases that match are highlighted in bold on the search results page.

Words, phrases, and modifiers
The basic search term is a word or a "phrase in quotes". Search recognizes a "word" to be A "stop word" is a word that is ignored (because it is common, or for other reasons).
 * a string of digits
 * a string of letters
 * subwords between letters/digit transitions, such as in txt2regex
 * subwords inside a compoundName using camel>w:CamelCase|camelCase

A given search term matches against content (rendered on the page). To match against wikitext instead, use the insource search parameter. Each search parameter has its own index, and interpret its given term in its own way.

Spacing between words, phrases, parameters, and input to parameters can include generous instances of whitespace and greyspace characters. "Greyspace characters" are all the non-alphanumeric characters ~!@#$%^&*_+-={}|[]\:";'<>?,./ . A mixed string of greyspace characters and whitespace characters, is "greyspace", and is treated as one big word boundary. Greyspace is how indexes are made and queries are interpreted.

Two exceptions are where 1) an embedded:colon is one word (it being treated as a letter), and 2) an embedded comma, such as in 1,2,3, is treated as a number Greyspace characters are otherwise ignored unless, due to query syntax, they can be interpreted as modifier characters.

The modifiers are ~ * ? - " ! . Depending on their placement in the syntax they can apply to a term, a parameter, or to an entire query. Word and phrase modifiers are the wildcard, proximity, and fuzzy searches. Each parameter can have their own modifiers, but in general
 * A fuzzy-word or fuzzy-phrase search can suffix a tilde ~ character (and a number telling the degree).
 * A tilde ~ character prefixing a query guarantees search results instead of any possible navigation.
 * A wildcard character inside a word can be a question ? mark for one character or an asterisk * character for more.
 * Truth-logic can interpret AND and OR, but parameters cannot.
 * Truth-logic understands - or ! prefixed to a term to invert the usual meaning of the term from "match" to "exclude".
 * Quotes around words mark an "exact phrase" search. For parameters they are also needed to delimit multi-word input.
 * Stemming is automatic but can be turned off using an "exact phrase".

A phrase search can be initiated by various hints to the search engine. Each method of hinting has a side-effect of how tolerant the matching of the word sequence will be. For greyspace, camelCase, or txt2number hints A "search instead" report is triggered when a universally unknown word is ignored in a phrase.
 * given words-joined_by_greyspace(characters) or wordsJoinedByCamelCaseCharacters it finds words joined by ... characters, in their bare forms or greyspace forms.
 * txt2number will match  or.
 * Stop words are enabled for the edge cases (in the periphery) of a grey_space or camelCase phrase. For example the_invisible_hand_of_a will match.

Each one of the following types of phrase-matching contains and widens the match-tolerances of the previous one: A word search will "additionally" find the words anywhere on the page.
 * An "exact phrase" "in quotes" will tolerate (match) with greyspace. Given "exact_phrase" or "exact phrase" it matches.
 * A greyspace_phrase initiates stemming and stop word checks.
 * Given CamelCase it will additionally match, all lowercase. CirrusSearch is case insensitive.

Some parameters interpret greyspace phrases, but other paramters, like insource only interpret the usual "phrase in quotes".

Note how the "exact phrase" search interpreted the embedded:colon character as a letter, but not the embedded_underscore character. A similar event occurs with the comma, character inside a number.

Given, CirrusSearch, when in an "exact phrase" context, (which includes the insource parameter context), will not match  ,  , or  , but will then only match.

Otherwise, remember that for CirrusSearch words are letters, numbers, or a combination of the two, and case does not matter.

The common word search employs the space character and is aggressive with stemming, and when the same words are joined by greyspace characters or camelCase they are aggressive with phrases and subwords.

When common words like "of" or "the" are included in a greyspace-phrase, they are ignored, so as to match more aggressively.

A greyspace_phrase search term, or a camelCase, or a txt2number term, match the signified words interchangeably. You can use any of those three forms.

Now camelcase matches camelCase because Search is not case sensitive, but camelCase matches camelcase because camelCase is more aggressive. Like the rest of Search, subword "words" are not case-sensitive. By comparison the "exact phrase" is greyspace oriented and ignores numeric or letter-case transitions, and stemming. "Quoted phrases" are not case sensitive.

From the table we can surmise that the basic search parser_function -"parser function" is the sum of the basic searches  and.

Making inquiries with numbers, we would find that
 * Plan9 or Plan_9 matches any of:,  ,  ,  ,
 * "plan9" only matches  (case insensitive)
 * Plan*9 matches  or.

The wildcard * character is basic and matches within a rendered word. It ignores the first (one or more) letters, then it can match a string of letters and digits. Or it ignores the first (one or more) digits, then it can match a sequence of zero or more numbers; capital letters, ordinal letters (st, nd, rd), or time abbreviations (am or pm); or parts of decimal numbers. For the wildcard character: Or *? is also accepted, but ?* is not recognized.
 * The comma is considered part of one number, but the decimal point is considered a greyspace character, and will delimit two numbers.
 * Inside an "exact phrase" it matches stemming plus compounding.
 * It may sometimes serve as an alternative to regex (an advanced search covered later);
 * Wildcards work for word, phrase, and insource searches, but not in the intitle parameter.
 * ? can represent one letter or number;

Putting a tilde ~ character after a word or phrase activates a fuzzy search.
 * For a phrase it is termed a proximity search, because proximal words are tolerated to an approximate rather than exact phrase.
 * For example, "exact one two phrase"~2 matches.
 * For a word it means extra characters or changed characters.
 * For an phrase a fuzzy search requires a whole number telling how many extra words to fit in, but
 * for a word a fuzzy search is a decimal fraction, defaulting to word~0.5 (word~.5), where at most two letters can be found swapped, changed, or added, but never the first two letters.
 * For a proximity phrase, a large number can be used, but that is an "expensive" (slow) search.
 * For a word word~.1 is most fuzzy, and word~.9 is least fuzzy, and word~1 is not fuzzy at all.

For the closeness value necessary to match in reverse (right to left) order, count and discard all the extra words, then add twice the the total count of remaining words minus one. (In other words, add twice the number of segments). For the full proximity algorithm, see Elastic Search. An explicit AND is required between two phrases because otherwise the two "inner" "quotation marks" are confused.

Quotes turn off stemming, "but appending"~ the tilde reactivates the stemming.

Quotes turn off stemming, "but appending"~ the tilde reactivates the stemming.

Insource
Insource searches can be used to find any one word rendered on a page, but it's made for finding any odd phrase you might find - including MediaWiki markup. This phrase completely ignores greyspace: insource: "state state autocollapse" matches.

Insource complements itself. On the one hand it has full text search for any word in the wikitext, instantly. On the other hand it can process a regexp search for any string of characters. Regex scan all the textual characters in a given list of pages; they don't have a word index to speed things up, and the process is interrupted if it must run more than twenty seconds. Regex run last, so to limit needless character-level scanning, you advance it a list of pages (a search domain) selected by a indexed search added to the query as a "clause", and you do this to every single regex query. . Insource can play both roles, and the best candidate for insource:/arg/ is often insource: arg, where arg is the same.

The syntax for the regexp is insource: no space, and then /regexp/. (No other parameter disallows a space. All the parameters except insource:/regexp/ generously accept space after their colon. )

Insource indexed-search and regexp-search roles are similar in many respects:
 * Both search wikitext only.
 * Neither finds things "sourced" by a transclustion>Wikipedia:transclusion|transclusion.
 * Neither does stemmed, fuzzy, or proximity searches.
 * Both want the fewest results, and both work faster when accompanied by another clause.

But indexed searches all ignore greyspace; wildcards searches do not match greyspace, so regex are the only way to find an exact string of any characters, for example a sequence of two spaces. Regex are an entirely different class of search tool that make matching a literal string in a regexp exact string search, a basic, easy search. Advanced regex are an entirely different endeavor than matching a literal string. See regexanchor>#Regular expression searches below.

Filters (intitle:, incategory: and linksto:)


We've tightened up the syntax around these quite a bit.


 * intitle:foo
 * Find articles whose title contains foo. Stemming is enabled for foo.
 * intitle:"fine line"
 * Find articles whose title contains fine line. Stemming is disabled.
 * intitle:foo bar
 * Find articles whose title contains foo and whose title or text contains bar.
 * -intitle:foo bar


 * Find articles whose title does not contain foo and whose title or text contains bar.
 * incategory:Music
 * Find articles that are in Category:Music
 * incategory:"music history"
 * Find articles that are in Category:Music_history


 * incategory:"musicals" incategory:"1920"
 * Find articles that are in both Category:Musicals and Category:1920
 * -incategory:"musicals" incategory:"1920"
 * Find articles that are not in Category:Musicals but are in Category:1920
 * cow*
 * Find articles whose title or text contains words that start with cow
 * linksto:Help:CirrusSearch
 * find articles that link to a page
 * -linksto:Help:CirrusSearch CirrusSearch
 * find articles that mention CirrusSearch but do not link to the page Help:CirrusSearch

prefix:
The prefix: syntax in its current form is relied upon for a great deal of functionality so it's been recreated as exactly as possible.

Note that the old rule of having to put prefix: at the end of the query still applies.

morelike:
Find articles about stinging insects. Find templates about regex searching for template usage on the wiki.
 * Find articles whose text is most similar to the text of the given articles.
 * Find articles whose text is most similar to the text of the given articles.

The  query works by choosing a set of words in the input articles and run a query with the chosen words. You can tune the way it works by adding the following parameters to the search results URL: These settings can be made persistent by overriding  in Help:System message.
 * : Minimum number of documents (per shard) that need a term for it to be considered.
 * : Maximum number of documents (per shard) that have a term for it to be considered.
 * : Maximum number of terms to be considered.
 * : Minimum number of times the term appears in the input to doc to be considered. For small fields this value should be 1.
 * : Minimal length of a term to be considered. Defaults to 0.
 * : The maximum word length above which words will be ignored. Defaults to unbounded (0).
 * (comma separated list of values): These are the fields to use. Allowed fields are,  ,  ,  ,   and.
 * ( | ): use only the field data. Defaults to : the system will extract the content of the   field to build the query.
 * : The percentage of terms to match on. Defaults to 0.3 (30 percent).
 * Example:

Namespace
A search domain consisting of one namespace, or "all", can be specified at the beginning of a query.

Two or more namespaces may be set at the search results page, Special:Search, in the Advanced dialog.

This can be set for the query, or for the user's default search domain.

Enter a namespace name, or enter, or enter a colon    for mainspace.

Namespace aliases are accepted.

For the File namespace,  is accepted.

Find articles in the Talk namespace whose title or text contains the word "foo".


 * Find articles in the File namespace on this wiki and commons whose title or text contains the word "foo".
 * Add  to the File namespace query to remove the results from commons.
 * Add  to the File namespace query to remove the results from commons.
 * Add  to the File namespace query to remove the results from commons.

You cannot use an interwiki prefix as a namespace to search other projects.

Did you mean
"Did you mean" suggestions are designed to notice if you misspell an uncommon phrase that happens to be an article title. If so, they'll let you know. They also seem to suggest more things than they ought to sometimes.

Prefer phrase matches
If you don't have too much special syntax in your query we'll give perfect phrase matches a boost. I'm being intentionally vague because I'm not sure exactly what "too much special syntax" should be. Right now if you add any explicit phrases to your search we'll turn off this feature.

Fuzzy search
Putting a ~ after a search term (but not double quotes) activates fuzzy search. You can also put a number from 0 to 1 to control the "fuzziness" fraction, e.g. nigtmare~.9 or lighnin~.1 or lighnin~0.1. Closer to one is less fuzzy.

Quotes and exact matches
Quotes turn on exact term matches. You can add a ~ to the quote to go back to the more aggressive matcher you know and love.

prefer-recent:
You can give recently edited articles a boost in the search results by adding "prefer-recent:" to the beginning of your search.

By default this will scale 60% of the score exponentially with the time since the last edit, with a half life of 160 days.

This can be modified like this: "prefer-recent:,".

proportion_of_score_to_scale must be a number between 0 and 1 inclusive.

half_life_in_days must be greater than 0 but allows decimal points.

This number works pretty well if very small.

I've tested it around .0001, which is 8.64 seconds.

This will eventually be on by default for Wikinews, but there is no reason why you can't activate it in any of your searches.

hastemplate:
You can find pages that use a certain template by adding the filter  to the search. We provide for the usual "syntactic sugar" of template calls. This means the lenient pagename and fullpagename capitalization works, and the main namespace abbreviation, ":" works. For example to find which pages transclude Quality image the full search (in all your preferred namespaces) can be: , and for that same template name in the main namespace, this works. You can omit the quotes if the template title does not contain a space. will filter pages that do not contain that template.

For wikitext that calls a template directly, you can use insource:, but hastemplate: searches the "post-expansion inclusion", so hastemplate: can find a template acting only temporarily as a "secondary template" or "meta-template", which are seen in neither the source nor content, ( but only included as a helper to any other template producing the final content). All content from a template is now reflected in search results is still the relevant philosophy here.

boost-templates:
You can boost pages' scores based on what templates they contain. This can be done directly in the search via  or you can set the default for all searches via the new   message. replaces the contents of  if the former is specified. The syntax is a bit funky but was chosen for simplicity. Some examples:


 * Find files in the China category sorting quality images first.
 * Find files in the China category sorting quality images first.


 * Find files in the China category sorting quality images first and low quality images last.
 * Find files in the China category sorting quality images first and low quality images last.


 * Find files about popcorn sorting quality images first and low quality images last. Remember that through the use of the  message this can be reduced to just.
 * Find files about popcorn sorting quality images first and low quality images last. Remember that through the use of the  message this can be reduced to just.

Don't try to add decimal points to the percentages. They don't work and search scoring is such that they are unlikely to matter much.

A word of warning about : if you add really really big or small percentages they can poison the full text scoring. Think, for example, if enwiki boosted featured articles by a million percent. Then searches for terms mentioned in featured articles would find the featured articles before exact title matches of the terms. Phrase matching would be similarly blown away so a search like  would find a featured article with those words scattered throughout it instead of the article for Brave New World.

Sorry for the inconsistent  in the name. Sorry again but the quotes are required on this one. Sorry also for the funky syntax. Sorry we don't try to emulate the template transclusion syntax like we do with.

insource:
This can pick up template arguments, URLs, links, html, etc.

It has two forms, one is an indexed search, and the other is regex based.

Tip: Instead of running a bare insource:/regexp/, these return much much faster when you limit the regexp search-domain to the results of one or more index-based searches. An "exact string" regexp search is a basic search; it will simply "quote" the entire regexp, or "backslash-escape" all non-alphanumeric characters in the string. All regexp searches also require that the user develop a simple filter to generate the search domain for the regex engine to search:



Any search without a namespace or prefix searches your default search domain, settable at Special:Search. It is commonly reset by power users to All namespaces, but if this occurs for a bare regexp search, then on a large wiki it will probably incur an HTML timeout before completing the search.

A regex search actually scours each page character-by character. By contrast, an indexed search actually queries a few records from a database separate from the wiki, and provides nearly instant results.

When using a regex, include other search terms to limit the regex search domain as much as possible.

There are many search terms that use an index and so instantly provide a highly refined search domain for the /regexp/. In order of general effectiveness:


 * insource:"" with quotation marks, duplicating the regexp except without the slashes or escape characters, is ideal.
 * intitle, incategory, and linksto are excellent filters.
 * "word1 word2 word3", with or without the quotation marks, are OK.
 * hastemplate: if it produces less than a few hundred thousand pages, is OK.
 * namespace: is practically useless, but may enable a slow regexp search to complete.

The prefix operator is especially useful with a { {FULLPAGENAME}} or a subdirectory argument.

To develop a new regexp, or refine a complex regexp, use  in any edit box.

Search terms that do not increase the efficiency of a regexp search are the page-scoring operators: morelike, boost-template, and prefer-recent.

Metacharacters
This section covers how to escape metacharacters.

For the actual meaning of the metacharacters see the [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html#regexp-syntax explanation of the syntax].

For the formal definition see the [http://lucene.apache.org/core/4_8_1/core/org/apache/lucene/util/automaton/RegExp.html Lucene grammar for regular expressions].

The use of a regexp to search for an exact string that includes non-alphanumeric characters is a basic search.

It finds regular expression metacharacters literally by placing the entire regexp inside double quotation marks, which blindly "quotes" or "escapes" any possible metacharacters from their advanced search meaning.

An advanced search usually escapes metacharacters one at a time with a backslash.

For example  matches a '2', a literal plus sign, another '2', an equals sign, a '4' and a literal dot, with one possible space character between each math term.

The equals sign has no special, metacharacter meaning in CirrusSearch, and so need not be escaped, but its OK to escape or quote any character because it basically has no effect.

An exact-string search usually "quotes" the regexp because this will neutralize all metacharacter meanings.

The square-bracket notation for creating your own character-class also escapes metacharacters.

To target a literal right square bracket in your character-class pattern, it must be backslash escaped, otherwise it can be interpreted as the closing delimiter of the character-class pattern definition.

The first position of a character class will also escape the right square bracket.

Inside the delimiting square brackets of a character class, the dash character also has special meaning (range) but can it too can be included literally in the class the same way as the right square bracket can.

For example both of these patterns target character that is either a dash or a right square bracket or a dot:  or.

Advanced example
For example, using metacharacters to find the usage of a template called Val having, inside the template call, an unnamed parameter containing a possibly signed, three to four digit number, possibly surrounded by space characters, AND on the same page, inside a template Val call, a named argument "fmt=commas" having any allowable spaces around it, (it could be the same template call, or a separate one):



It is fast because it uses two filters so that every page the regexp crawls has the highest possible potential.

Assuming your search domain is set to ALL, it searches the entire wiki, because it offers no namespace or prefix.

bounded
You can limit search to pages identified as being near specified geographic coordinates. The coordinates can either be specified as a, pair, or by providing a page title from which to source the coordinates. A distance to limit the search to can be prepended if desired. Examples:


 * neartitle:"San Francisco"
 * neartitle:"100km,San Francisco"
 * nearcoord:37.77666667,-122.39
 * nearcoord:42km,37.77666667,-122.39

boosted
You can alternatively increase the score of pages within a specified geographic area. The syntax is the same as bounded search, but with boost- prepended to the keyword. This effectively doubles the score for pages within the search range, giving a better chance for nearby search results to be near the top.


 * boost-neartitle:"San Francisco"
 * boost-neartitle:"100km,San Francisco"
 * boost-nearcoord:37.77666667,-122.39
 * boost-nearcoord:42km,37.77666667,-122.39

Auxiliary text
Cirrus considers some text in the page to be "auxiliary" to what the page is actually about. Examples include table contents, image captions, and "This article is about the XYZ. For ZYX see ZYX" style links. You can also mark article text as auxiliary by adding the  class to the html element containing the text.

Auxiliary text is worth less than the rest of the article text and it is in the snippet only if there are no main article snippets matching the search.

Lead text
Cirrus assumes that non-auxiliary text that is between the top of the page and the first heading is the "lead in" paragraph. Matches from the lead in paragraph are worth more in article ranking.

Commons search
By default when the search contains the file namespace, Cirrus will search commons as well.

You can disable this behavior by adding  to the search.

If you are using a namespace prefix to select the namespace the syntax looks like.

If you aren't using a namespace prefix to select the namespace then the syntax looks like.