User:EBernhardson (WMF)/Notes/Accept-Language
November 2015 - Questions and comments are welcome on the talk page
- This is currently a work in progress and is not complete*
Hypothesis[edit]
Using the first non-English Accept-Language HTTP header will provide a good proxy for the language the query is in when the query returns no results against the wiki it was already run against. Further that this is a better proxy than the existing elasticsearch langdetect plugin.
Final Results[edit]
180,298 full text desktop queries to enwiki for the hour
28,929 filtered to those with a usable non-English accept-language header
2,015 number of those queries that give zero results
395 number of those queries which convert to non-zero results via accept-language
21% conversion rate of zero result with usable accept-language header to non-zero result
21,022 Estimated number of full text desktop queries to enwiki for the hour that have zero results
1.9% conversion rate from zero result to having result for full data set
Comparison to language detector:
2,015 number of zero result queries from above
1,429 number of those that detect to non-english
128 number of those which convert to non-zero results via language detection
9% conversation rate of zero result with usable accept-language header to non-zero result via lang-detect
Generally this looks to imply we should prefer the accept-language header over language detection for choosing a second wiki to query. We will still need to use language detection as only 16% of queries had a usable accept-language header.
Caveats[edit]
This only analyzed traffic to enwiki. It is likely if we ran this for traffic going to ruwiki or zhwiki we would get slightly different results. Additionally this only considers desktop search. Query rewriting for the API is not done by default, but is instead hidden behind a feature flag. As such even if we could have an effect on API requests most of them do not enable the rewrite flag and will not be affected. Note that the API makes up something like 75% of all search requests.
Additionally this data set, once filtered to queries with non-English accept headers, has a zero result rate of only 7%. This is 40% lower ((11.6-6.9)/11.6) than the overall zero result rate recorded in our CirrusSearchRequestSet logs for the same hour. Not sure if this means anything, but seems like a large variance.
Process[edit]
Extract one hour worth of desktop full text searches from webrequest logs[edit]
Started by taking an hour worth of queries + accept language headers for enwiki from hive wmf.webrequests table using the following query. The specific day and hour to work with was arbitrarily chosen. This gives us a set of 180,298 queries to start with, which we will use to calculate the expected change to zero result rate. This feels much too low to be the total number of full text queries on enwiki for that hour, but is probably a reasonable number to run this test against.
INSERT OVERWRITE LOCAL DIRECTORY '/home/ebernhardson/hive'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
SELECT accept_language, query_string['search']
FROM (SELECT accept_language, str_to_map(substr(uri_query, 2), '&', '=') as query_string
FROM wmf.webrequest
WHERE year = 2015 AND month = 11 AND day = 04 AND hour = 04
AND uri_host = 'en.wikipedia.org'
AND uri_path <> '/w/api.php') x
WHERE length(query_string['search']) > 0
AND (query_string['title'] = 'Special:Search' OR query_string['title'] = 'Special%3ASearch');
Filter out requests with only English or invalid accept-language[edit]
The result of the above query was then run through the following php script to filter out queries that had an invalid accept-language header recorded, or only included English. Run against the above set of 180,298 queries we end up with 28,929 queries that could be affected. This means around 16% of our search queries to enwiki contain a non-English Accept-Language header.
<?php
$data = array();
while( false !== ( $line = fgets( STDIN ) ) ) {
list( $acceptLang, $term ) = explode( "\t", rtrim( $line, "\n" ) );
$term = trim( $term );
if ( strlen( $term ) === 0 ) {
continue;
}
$parsed = parseAcceptLang( $acceptLang );
foreach ( array_keys( $parsed ) as $lang ) {
if ( substr( $lang, 0, 2 ) !== 'en' ) {
$data[] = array( $acceptLang, $term );
break;
}
}
}
usort( $data, function( $a, $b ) {
return strcmp( $a[1], $b[1] );
} );
foreach ( $data as $value ) {
echo implode( "\t", $value ) . "\n";
}
Run the queries through enwiki to find which are zero result queries[edit]
These queries are then run against the enwiki index we have in the hypothesis-testing cluster to see which are zero result queries. This was done with the following command line:
cat queries.accept-lang.sorted | \
ssh -o Compression=yes suggesty.eqiad.wmflabs 'sudo -u vagrant mwscript extensions/CirrusSearch/maintenance/runSearch.php --wiki=enwiki --decode --options='\''{"wgCirrusSearchEnableAltLanguage":false}'\' | \
pv -l -s $(wc -l queries.accept-lang.sorted | awk '{print $1}') \
> queries.accept-lang.sorted.results
The results of this were filtered down to only the zero result queries, giving only 2,015 queries to test our original hypothesis against. Note that 2,015 out of 28,929 means this set had a zero result rate of 6.9% which is much lower than expected. I queried the same hour from the CirrusSearchRequestSet in hive and came up with a ZRR of 11.7% (see appendix for the query used).
cat queries.accept-lang.sorted.results | \
jq -c 'if (.totalHits > 0) . else empty end' |\
wc -l
Sort zero result queries into a bucket per target wiki[edit]
For the next step I needed a map from the languages to their wikis. This was sourced from this gerrit patch.
These queries were then separated out into a file per wiki using the following php script.
<?php
$langMap = include __DIR__ . '/langs.php';
$queryFile = fopen( $argv[1], "r" );
$resultFile = fopen( $argv[2], "r" );
$match = $total = $zeroResult = $hasAcceptLang = $error = 0;
while ( !feof( $queryFile ) && !feof( $resultFile ) ) {
$line = rtrim( fgets( $queryFile ), "\n" );
list( $accept, $encodedQuery ) = explode( "\t", $line, 2 );
$query = urldecode( $encodedQuery );
// not sure why this is necessary, there is some sort of bug in runSearch.php
// most likely, but a quick review didn't turn anything up.
while ( $query === "0" ) {
$line = rtrim( fgets( $queryFile ), "\n" );
list( $accept, $encodedQuery ) = explode( "\t", $line, 2 );
$query = urldecode( $encodedQuery );
}
$result = json_decode( rtrim( fgets( $resultFile ), "\n" ), true );
$total++;
if ( $query !== $result['query'] ) {
continue;
}
$match++;
if ( isset( $result['error'] ) ) {
$error++;
continue;
}
// totalHits will be not set for empty queries, such as ' '
if ( !isset( $result['totalHits'] ) || $result['totalHits'] > 0 ) {
continue;
}
$zeroResult++;
// we now have a query and know it's a zero result, strip the accept
// language header down to the first accepted language that is not
// english
$parsedAcceptLang = array_keys( parseAcceptLang( $accept ) );
$tryWiki = null;
foreach ( $parsedAcceptLang as $lang ) {
$shortLangCode = preg_replace( '/-.*$/', '', $lang );
if ( isset( $langMap[$shortLangCode] ) ) {
$tryWiki = $langMap[$shortLangCode];
break;
}
}
if ( $tryWiki === null ) {
continue;
}
file_put_contents( $tryWiki, urlencode( $query ) . "\n", FILE_APPEND );
$hasAcceptLang++;
}
fwrite( STDERR, "\nMatch: $match\nTotal: $total\nError: $error\nZero Result: $zeroResult\nHas accept-language: $hasAcceptLang\n");
A quick look at which wikis the queries were assigned to was done with
wc -l * | sort -rn | head -n 21 | tail -n 20
This gives the following top 20 targets of queries from enwiki in the analyzed hour:
545 zhwiki 329 kowiki 241 svwiki 220 eswiki 114 jawiki 53 dewiki 51 ruwiki 50 arwiki 45 thwiki 44 ptwiki 43 frwiki 33 hiwiki 32 idwiki 19 viwiki 17 mswiki 14 hewiki 13 plwiki 12 nlwiki 11 fiwiki 10 trwiki
I don't have the resources available to run the full 180k query set to calculate it's ZRR rate, but we can estimate using #Calculate_zero_result_rate_from_hive_CirrusSearchRequestSet_table. This gives a ZRR of 11.7%, which suggests 21k of the 180k queries would have zero results. We were able to convert 395 of those 21k queries into results for a conversion rate of 1.9%
Search target wikis[edit]
Now that we have all the zero result queries that have a usable accept-language header broken out into files per wiki we can run them with the following:
for i in $(wc -l * | sort -rn | head -n 21 | tail -n 20 | awk '{print $2}'); do
cat $i | \
ssh -o Compression=yes suggesty.eqiad.wmflabs 'sudo -u vagrant mwscript extensions/CirrusSearch/maintenance/runSearch.php --wiki='$i' --decode --options='\''{"wgCirrusSearchEnableAltLanguage":false}'\' | \
pv -l -s $(wc -l $i | awk '{print $1}') \
> $i.results
done
These can then be processed to get the new zero result rate:
(for i in *.results; do
TOTAL="$(wc -l < $i)"
NONZERO="$(cat $i | jq -c 'if (.totalHits > 0) then . else empty end' | wc -l)"
echo $i total: $TOTAL non-zero percent: $(echo "scale=3; $NONZERO / $TOTAL" | bc)
done) | sort -rnk3
Which results in:
zhwiki.results total: 545 non-zero percent: .216 kowiki.results total: 329 non-zero percent: .079 svwiki.results total: 241 non-zero percent: .522 eswiki.results total: 220 non-zero percent: .222 jawiki.results total: 114 non-zero percent: .140 dewiki.results total: 53 non-zero percent: .207 ruwiki.results total: 51 non-zero percent: .235 arwiki.results total: 50 non-zero percent: .120 thwiki.results total: 45 non-zero percent: .066 ptwiki.results total: 44 non-zero percent: .136 frwiki.results total: 43 non-zero percent: .162 hiwiki.results total: 33 non-zero percent: 0 idwiki.results total: 32 non-zero percent: .281 viwiki.results total: 19 non-zero percent: 0 mswiki.results total: 17 non-zero percent: .058 hewiki.results total: 14 non-zero percent: 0 plwiki.results total: 13 non-zero percent: .076 nlwiki.results total: 12 non-zero percent: .083 fiwiki.results total: 11 non-zero percent: .090 trwiki.results total: 10 non-zero percent: .200
Across the top 20 wikis there are 1896 queries, and 395 were able to find results. This gives an overall 20.8% conversion rate. Considering the full set of queries, including those against languages we did not run due to being in the long tail, we have 395/2011 = 19.6%.
Compare against language detection[edit]
To determine if language detection does a better job than the accept-language header we take the set of 1896 queries that were made above and re-bucket them based on language detection. That was done with the following code:
$errors = $undetectable = $detected = $unknown = 0;
while ( false !== ( $line = fgets( STDIN ) ) ) {
$encodedQuery = rtrim( $line, "\n" );
$lang = detectLanguage( $encodedQuery );
if ( $lang === null ) {
$undetectable++;
continue;
}
$lang = preg_replace( '/-.*$/', '', $lang );
if ( isset( $langs[$lang] ) ) {
file_put_contents( $langs[$lang], $line, FILE_APPEND );
$detected++;
} else {
// usually 'en'
$unknown++;
}
}
fwrite( STDERR, "Errors: $errors\nUndetectable: $undetectable\nUnknown: $unknown\nDetected: $detected\n" );
// taken from CirrusSearch\Searcher::detectLanguage()
function detectLanguage( $encodedText ) {
$ch = curl_init( 'http://localhost:9200/_langdetect' );
curl_setopt( $ch, CURLOPT_POST, true );
curl_setopt( $ch, CURLOPT_POSTFIELDS, $encodedText );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
$value = json_decode( curl_exec( $ch ), true );
if ( $value && !empty( $value['languages'] ) ) {
$langs = $value['languages'];
if ( count( $langs ) === 1 ) {
return $langs[0]['language'];
}
if ( count( $langs ) === 2 ) {
if ( $langs[0]['probability'] > 2*$langs[1]['probability'] ) {
return $langs[0]['language'];
}
}
}
return null;
}
It was run as follows:
mkdir per-wiki.lang-detect
cd per-wiki.lang-detect
cat ../per-wiki.accept-lang/*wiki | php ../lang-detect.php
Re-running the same analysis as above for per-wiki ZRR we get:
ptwiki.results total: 193 non-zero percent: .015 itwiki.results total: 149 non-zero percent: .120 rowiki.results total: 148 non-zero percent: .081 dewiki.results total: 143 non-zero percent: .118 tlwiki.results total: 87 non-zero percent: 0 frwiki.results total: 67 non-zero percent: .119 eswiki.results total: 65 non-zero percent: .307 sqwiki.results total: 57 non-zero percent: 0 idwiki.results total: 53 non-zero percent: .207 huwiki.results total: 47 non-zero percent: 0 svwiki.results total: 40 non-zero percent: .350 ltwiki.results total: 38 non-zero percent: 0 nowiki.results total: 36 non-zero percent: .138 zhwiki.results total: 31 non-zero percent: .129 dawiki.results total: 31 non-zero percent: .032 hrwiki.results total: 29 non-zero percent: .034 fiwiki.results total: 29 non-zero percent: .034 plwiki.results total: 28 non-zero percent: .071 etwiki.results total: 23 non-zero percent: 0 trwiki.results total: 22 non-zero percent: .045 nlwiki.results total: 20 non-zero percent: .250 viwiki.results total: 12 non-zero percent: 0 lvwiki.results total: 7 non-zero percent: 0 cswiki.results total: 7 non-zero percent: .142
This finds results for 124 queries out of 2015, for a conversion rate of 6.1%.
Additional functions used in above scripts[edit]
parseAcceptLang[edit]
// sourced from mediawiki WebRequest::getAcceptLang()
function parseAcceptLang( $acceptLang ) {
if ( !$acceptLang ) {
return array();
}
$acceptLang = strtolower( $acceptLang );
$lang_parse = null;
preg_match_all(
'/([a-z]{1,8}(-[a-z]{1,8})*|\*)\s*(;\s*q\s*=\s*(1(\.0{0,3})?|0(\.[0-9]{0,3})?)?)?/',
$acceptLang,
$lang_parse
);
if ( !count( $lang_parse[1] ) ) {
return array();
}
$langcodes = $lang_parse[1];
$qvalues = $lang_parse[4];
$indices = range( 0, count( $lang_parse[1] ) - 1 );
foreach ( $indices as $index ) {
if ( $qvalues[$index] === '' ) {
$qvalues[$index] = 1;
} elseif ( $qvalues[$index] == 0 ) {
unset( $langcodes[$index], $qvalues[$index], $indices[$index] );
}
}
array_multisort( $qvalues, SORT_DESC, SORT_NUMERIC, $indices, $langcodes );
return array_combine( $langcodes, $qvalues );
}
Calculate zero result rate from hive CirrusSearchRequestSet table[edit]
SELECT SUM(results.outcome) AS non_zero,
COUNT(*) - SUM(results.outcome) AS zero,
1 - SUM(results.outcome) / COUNT(*) AS zero_result_rate
FROM ( SELECT IF(array_sum(requests.hitstotal) > 0, 1, 0) AS outcome
FROM ebernhardson.cirrussearchrequestset
WHERE year=2015 AND month=11 AND day=4 AND hour=4
AND wikiid='enwiki'
AND source='web'
AND array_contains(requests.querytype, 'full_text')
) AS results;
Result:
non_zero zero zero_result_rate 131003 17298 0.11664115548782539