This page describes the Wikimedia Foundation's activities surrounding our sites' search functionality.
The Wikimedia search infrastructure hasn't had significant development work for many years. The current system is based on homegrown layer on top of Lucene (lsearchd) that has since been tackled by much larger projects such as Solr. The search system frequently breaks in ways that are difficult to diagnose, and generally makes our Operations staff sad.
Goals for our current effort:
- Make our existing tools more robust
- Improve logging in our existing tools to make problems easier to diagnose
- Migrate away from lsearchd to Solr (or something similar)
Our current search infrastructure is highly outdated and difficult to manage due to tons of custom code. We'd like to replace it with Solr (also based on Lucene), as it's very stable, contains many of the features we need, and doesn't require nearly as much custom code to support.
Solr implementation plan
We don't yet have a firm timeline for a Solr migration. A few considerations.
- Solr is web-based and has its own query syntax (Solr query syntax)
- We have a rather complex set of search modes that we support in lsearchd (user documentation)
- As an initial step, we need to decide how much of the lsearchd syntax we want to support in the Solr implementation and if we want to enhance it in some way to take advantage of newer Solr capabilities (e.g. RegEx search). This will have a strong impact on the rest of the architecture since it determines what indices are generated.
- Based on this, we need to map out how MWSearch extension needs to change for Solr.
- For a new implementation, some sort of incremental approach seems best where we deploy Solr for smaller wikis first, and learn from that experience for the larger wikis.
- A solid PHP library
- Translation memory and GeoData both use Solarium, which is widely used and very robust.
- Solr library in Pecl is poorly maintained, incomplete.
For core search
For other WMF applications
The index is relatively small (so no need to make it distributed), but requires a lot of computational power to work with. Full-text search is not currently used. Currently, data from all the wikis is stored in the same core, in the future we will need to split data to many cores (the puppet changes for using multiple cores with shared configuration/schema are here, needs more work).
- Load expectations: unclear, but will be high if we start using it heavily e.g. for maps display.
- Backups: not really needed - if master is down just switch to a slave. If all servers are down, reindexing from scratch is quick.
- Note: because GeoData's schema is very stripped-down, /admin/ping doesn't work - should be remebered if someone wants to rewrite the current monitoring.
Nice to haves
- A pony
- Product tasks
- Help prioritize the bug reports on Search:
- Ram -- looking at bugzilla:43663
- Search documentation on Wikitech: wikitech:Search
- Ram's setup instructions: wikitech:User:Ram/Search
- Some notes from Brion in 2008
- The MWSearch extension provides a SearchEngine subclass which contacts Wikimedia's Lucene-based search server. This replaces the older LuceneSearch extension which reimplemented the entire Special:Search page.
- /2013-02 discussion - Discussion with Rainman about how the current system works