Extension:SphinxSearch

Description
As MediaWiki-based site administrators, one of the most common complaints we receive is that the default search engine is far from excellent. In our day and age where Google sets the standard for search engine capabilities, users aren't happy with a basic search engine. They need, or should I say demand a faster, easier, better engine.

The Sphinx Search Engine seems to promise exactly that; a full text search engine that is both flexible and fast. This extension incorporates the Sphinx engine into MediaWiki to provide a better alternative for searching. The extension can be installed in one of two modes:
 * 1) Provide an additional Special Page for searching using Sphinx. This method is excellent for providing a method for evaluating the performance of the extension while still maintaining the default search engine.
 * 2) Completely replace the built in search engine with the sphinx search engine.

This extension is very similar in nature to Extension:LuceneSearch. The main difference is obviously the search engine backend.

Compatibility
This extension has been shown to work / not work with the following MediaWiki versions. Please add more successes and failures to this list
 * 1.9.3 - Works - (Gri6507)
 * 1.10 - Works - (Svemir)
 * 1.11 - Works - (125.17.142.146)

The extension has been shown to work / not work with the following Sphinx versions. Please add more successes and failures to this list
 * 0.9.7 - Works - (Gri6507)
 * 0.9.8rc1 - Does not work - (125.17.142.146)

Step 1 - Install Sphinx
Download Sphinx Search Engine. Follow the installation instructions. You only need to do the actual installation, which means you do not need to do the "Quick Sphinx usage tour". You can verify your installation by following the rest of the steps here.

Step 2 - Configure Sphinx
Download and extract the extension to a temporary directory. Copy the sphinx.conf file from this download to some directory (we will refer to this file as "/path/to/sphinx.conf" below.) This directory should not be web-accessible, so you should not use the extensions folder. Make sure to adjust all values to suit your setup:
 * Set correct database, username, and password for your MediaWiki database
 * Update table names in SQL queries if your MediaWiki installation uses a prefix
 * Update the file paths (/var/data/sphinx/..., /var/log/sphinx/...) and create folders as necessary
 * If your wiki is very large, you may want to consider specifying a query range in the conf file.
 * If your wiki is not in English, you will need to change (or remove) the morphology attribute.

Note: To give credit where credit is due, we must thank the author of this excellent article for providing an excellent starting point on configuring this file.

Step 3 - Run Sphinx Indexer
Run the sphinx indexer to prepare for searching: Once again, make sure to replace the paths to match your installation. This process is actually pretty fast, but clearly depends on how large your wiki is. Just be patient and watch the screen for updates.

Step 4 - Test Out Sphinx
When the indexer is finished, test that sphinx searching is actually working: You will see the result stats immediately (Sphinx is FAST.) Note that the article data you see at this point comes from the sql_query_info in sphinx.conf file. In the extension we can get to the actual article content because we have text old_id available as an extra attribute. It would be slow to fetch article content on the command line (we would have to join page, revision, and text tables,) so we just fetch page_title and page_namespace at this point.

Step 5 - Start Sphinx Daemon
In order to speed up the searching capability for the wiki, we must run the sphinx in daemon mode. Add the following to whatever sever startup script you have access (i.e. /etc/rc.local): Note: without the daemon running, searching will not work. That is why it is critical to make sure the daemon process is started every time the server is restarted.

Step 6 - Configure Incremental Updates
To keep the index for the search engine up to date, the indexer must be scheduled to run at a regular interval. Setup a cron job for the full index - for example once every night: 0 3 * * * /path/to/sphinx/installation/indexer --config /path/to/sphinx.conf wiki_main --rotate > /dev/null 2>&1 Setup a more frequent cron to update the smaller index regularly: 0 9,15,22 * * * /path/to/sphinx/installation/indexer --config /path/to/sphinx.conf wiki_incremental --rotate > /dev/null --rotate 2>&1

As before, make sure to adjust the paths to suit your configuration. Note that --rotate option is needed if searchd deamon is already running, so that the indexer does not modify the index file while it is being used. It creates a new file and copies it over the existing one when it is done.

Step 7 - Extension Preparation
Copy the Sphinx API file, sphinxapi.php to the extensions directory. This file is part of the sphinx source code, under the api/ directory.

Step 8 - Extension Preparation
Download ExtensionFunctions.php from SVN and copy it to the extensions directory.

Step 9 - Extension Installation
Copy SphinxSearch.php and SphinxSearch_body.php from the temporary directory you extracted the code to in to your extensions directory.

Step 10 - Extension Installation
Add the following text to your LocalSettings.php

Options
For the most part, the extension's default options do not need any modification. However, if tweaking is needed/desired, there are a number of configuration options that could be configured from LocalSettings.php or from SphinxSearch.php directly. Those are:
 * $wgSphinxSearch_host - the hostname on which sphinxd is running (default to localhost)
 * $wgSphinxSearch_port - the port number on which sphinxd is running (default to 3312)
 * $wgSphinxSearch_mode - the Sphinx search mode. The default mode is the most intuitive. See Sphinx documentation for other valid options.
 * $wgSphinxSearch_matches - the number of search hits to display per result page.
 * $wgSphinxSearch_weights - the way Sphinx orders the results. The default is pretty good. See Sphinx documentation for other valid options.
 * $wgSphinxSearch_groupby, $wgSphinxSearch_groupsort - define how to group the results. See Sphinx documentation for other valid options.
 * $wgSphinxSearch_sortby - set matches sorting mode (default to SPH_SORT_RELEVANCE). See Sphinx documentation for other valid options.

When setting these options in LocalSettings.php, make sure to do so after the call to require_once for this extension.

Mode Of Operation
By default, this extension will run so as not to overwrite the built-in search engine, but instead provide a new Special Page called Search Wiki Using Sphinx. This allows the users to evaluate this extension by directory comparing the search results of the built-in search vs. Sphinx search.

If the performance is deemed acceptable to replace the built-in search engine, this extension can easily be configured to act as the default search engine. To do some, modify SphinxSearch.php to uncomment the lines containing Now, the standard search method, will use Sphinx by default. Note: when used in this way, the extension preseves the functionality of the Go and Search buttons.

ToDo

 * We use SPH_MATCH_EXTENDED for better relevance weights, but we process the search term to make it assume an OR instead of an AND on multiple. This will be replaced with an option on the search form. Done!
 * Due to the way the weights are calculated, it is hard to get title matches to always appear first. That can be solved by internally running the search twice, first time with @page_title attribute, second time with @old_text.
 * Sort the results in SPH_SORT_EXTENDED mode by @relevance and by number of times the page has been viewed (available from wiki database). The idea behind this is that given two pages that have the same relevance to the search, if one has been viewed more times, there is probably a reason for that.
 * try to parse the wiki text around the matching words and display in HTML instead of wiki format. This could be difficult because
 * the search matches could be inside wiki text that actually gets rendered as HTML that does not contain the same matches
 * the the sphinx method of pulling out text near the matched word may produce results that are not in valid wiki format (i.e. drop the leading [[, or the trailing ''' ).
 * Add the "did you mean" functionality to the search results. (This is in CVS already, some more tweaking required)

Revisions

 * v0.4.3 - October 15, 2007 - incremental bug fix. When sphinx is not the default search engine, viewing pages 2 and up of the results now actually uses sphinx.
 * v0.4.2 - October 12. 2007 - incremental bug fix. When sphinx is not the default search engine, the special page search actually uses sphinx now.
 * v0.4.1 - October 11, 2007 -
 * Made it optional to replace the default search with Sphinx completely. By default, Sphinx search becomes just another special page.
 * Fixed a bug when search would crash if a matching article was deleted after last indexer run.
 * v0.3 - October 5, 2007 - Numerous updates and improvements by Svemir Brkic
 * v0.1 - September 24, 2007 - Initial release (RFC)