Wikimedia Discovery

The Discovery department of Wikimedia Engineering is building the anonymous path of discovery to a trusted and relevant source of knowledge. We have a number of projects detailed below that fit into an initiative named "Knowledge Engine" that focuses us on creating and supporting new forms of discovery for our users. There is no one single product that fits under the Knowledge Engine roadmap but instead it's a collection of projects that focus on improving language/interwiki search, wikipedia.org portal(s), and new initiatives such as importing new data sources like Maps and query infrastructure like WDQS.

You can find all of our data and key performance indicators on our data portal.

Search
Discovery is responsible for maintaining and enhancing the various Search features and APIs for MediaWiki. This includes the CirrusSearch extension which relies on Elasticsearch, the search backend used at Wikimedia.

Current (FY 2015-16 Q2) Goal (link):
 * Improve language support for search.

Current work by this team is tracked on this Phabricator workboard.

Wikipedia.org portal
Many users discover Wikimedia via https://www.wikipedia.org/, so Discovery will be looking at how to improve the user experience on that page. Here is an initial analysis.

Current (FY 2015-16 Q2) Goal (link):
 * Make www.wikipedia.org a portal for exploring open content on Wikimedia sites.

Maps
Discovery is about finding and navigating to content, and one way for users to do that is via maps. To provide better maps, OpenStreetMap/Production maps cluster is a project to make OpenStreetMap available on all Wikimedia projects, at a scale sufficient for their widespread usage. The main sub-project page is here: Maps.

Work by this team is tracked on this Phabricator workboard

Wikidata Query Service (WDQS)
Searching structured data on Wikidata is also part of Discovery, so we are building the Wikidata query service. It provides a SPARQL API through which tools can access Wikidata.

Current work by this team is tracked on this Phabricator workboard

Analysis
The analysis group within Discovery manages the Discovery Dashboard, as well as analyzing A/B tests and other data.

Current (FY 2015-16 Q2) Goals (link):
 * Improve understanding of user satisfaction for search by iterating and improving on the search satisfaction metric.
 * Support ongoing evaluation of usage of Wikidata Query and Maps services to decide on what's next for these services.
 * Create recurring performance indicator on referrer traffic from the primary search engines and determine what features may have largest impact on referrer metric

Current work by the analysis team is tracked on this Phabricator workboard

APIs
API:Search and discovery lists the search APIs available and in development.

Members

 * Tomasz Finc, Head of Discovery, Director
 * Dan Garry, Lead Product Manager
 * Moiz Syed, Design Manager
 * Julien Girault, User Experience Engineer
 * Oliver Keyes, Data Analyst
 * Yuri Astrakhan, Senior Software Engineer
 * Erik Bernhardson, Software Engineer
 * Stas Malyshev, Senior Performance Engineer
 * Max Semenik, Software Engineer
 * Kevin Smith, Agile Coach
 * David Causse, Software Engineer
 * Trey Jones, Software Engineer
 * Mikhail Popov, Data Scientist

Roles and responsibilities
Roles and responsibilities for team members other than developers can be found here. The short form is:


 * Director: "Managing People and Coordinating w/Engineering"
 * Product Manager: "Product Vision and Story Prioritization"
 * UX Design: "UX Design and Vision, and leading UI engineers"
 * Engineering Team Lead: "Architecture and Code Quality"
 * Engineer:
 * Agile Coach: "Facilitation and Process Improvement"

We are hiring. See https://wikimediafoundation.org/wiki/Work_with_us

Mailing list
Discovery

Twitter
https://twitter.com/WMF_Discovery

Meetup groups

 * San Francisco
 * Directly relevant
 * Bay Area NLP (Natural Language Processing, not Neuro-Linguistic Programming)
 * San Francisco text
 * Elasticsearch San Francisco
 * Indirectly-related (these sorts of Meetup groups attract smart/enthusiastic people who like to spend their free time learning and solving problems)
 * Silicon Valley Java user group
 * San Francisco PHP
 * Bay Area Haskell users group
 * Scala study group
 * SF JavaScript
 * Oakland advanced Scala study group

Upcoming events

 * Wikimedia Developer Summit 2016
 * San Francisco January 4-6, 2016
 * https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016
 * WMF All Hands 2016
 * San Francisco January 7 and 8, 2016
 * https://office.wikimedia.org/wiki/All_hands/2016
 * Elastic{ON} 2016
 * San Francisco February 17-19, 2016
 * "the largest gathering of Elasticsearch, Logstash, and Kibana expertise anywhere in the world."
 * https://www.elastic.co/elasticon

Past Events

 * Hackathon 2015
 * Lyon, France, May.
 * OpenAir 2015
 * June 4 in San Francisco.
 * https://openair2015.com/
 * "OpenAir is the premier conference that focuses on creating engineering solutions to the challenges of matching. The brightest minds in the industry will come together to tackle such issues as search and discovery, trust, internationalization, mobile, identity, and infrastructure."
 * State of the Map US 2015
 * June 6-8 in New York.
 * An annual conference for all OpenStreetMap users. http://stateofthemap.us/
 * Yuri & Max attended
 * Wikimania 2015 (July 15-19 in Mexico City)
 * Presentation (video) "Are we failing our users when they search Wikipedia?" by Dan and Moiz
 * http://wikimania2015.wikimedia.org/wiki/Main_Page
 * Smart Data Conference 2015 (August 18-20 in San Jose, CA)
 * http://smartdata2015.dataversity.net/
 * Presentation (hosted at blazegraph.com): "The Wikidata Query Service - A Knowledge Graph Application Powered by Blazegraph"
 * Gerrit Cleanup Day
 * Wednesday 2015-09-23
 * Plans here: Discovery plans for gerrit cleanup day 2015
 * WikiConference USA
 * Washington DC, October 9-11, 2015
 * http://wikiconferenceusa.org/wiki/2015/Main_Page
 * At least one team member will attend. No presentations planned.
 * Discovery offsite 2015
 * Sept 30-Oct 2, Cocoa Beach FL USA
 * 5th DBpedia Community in California 2015
 * Palo Alto November 5th
 * "DBpedia is a crowd-sourced community effort to extract structured information from Wikipedia and make this information available on the Web."
 * http://wiki.dbpedia.org/meetings/California2015

Elasticsearch cluster

 * How Elasticsearch breaks Part 1 Part 2
 * Notes on unbreaking and optimizing elasticsearch

Meeting minutes

 * Weekly checkins:
 * Discovery/Checkin meeting minutes
 * Retrospectives
 * Team retrospective 2015-05-06
 * Lyon Hackathon
 * Team retrospective 2015-06-16
 * Team retrospective 2015-07-13
 * Team retrospective 2015-08-10
 * Team retrospective 2015-09-14
 * Team retrospective 2015-10-14
 * Team retrospective 2015-11-02
 * Quarterly reviews
 * Q4 2014-15 (2015-07-07)
 * Q1 2015-16 (2015-10-05)
 * Other
 * Maps and KPIs 2015-09-14

Deployers
Useful reference for who can deploy code. Its nice to know whom to bug if you need something: