Wikimedia Developer Summit/2017/Integrating MediaWiki (and other services) with dynamic configuration

Session Overview
Title: Integrating MediaWiki (and other services) with dynamic configuration

Day & Time: Tuesday Jan. 10th, 11:00 am – 12:10 pm (19:00 – 20:10 UTC)

Room: Hawthorne

Phabricator Task Link: https://phabricator.wikimedia.org/T149617

Facilitator(s): paravoid, greg-g

Note-Taker(s): Volans, bd808

Remote Moderator:

Advocate:

Detailed Summary

 * Detailed summary


 * Session slides: https://docs.google.com/presentation/d/1r4g_3yZxfPzzEEvOgj4ZbbTBrNshBylDAb5KYdnzke4/edit?usp=sharing

Purpose
Find a unified, simple and reliable way to manage the dynamic configuration (live state) of services in WMF.

Agenda
40 minutes presentation, 30 minutes discussions and conclusions

Style
Consensus: coming to agreement on one solution

Discussion Topics
The current way of managing the state of a cluster in WMF is via it's configuration and each change requires a change in the configuration repository (Puppet).

The live state of a service or cluster should be managed separately in a simple and reliable way that allows to change the state quickly, reliably and without running Puppet.

The main two categories of live state are:


 * Service discovery (SOA address book)
 * State management

Two solutions were proposed, based on the different use case:


 * DNS for simple data structures and service discovery through TXT or URI records
 * Confd for more complex data structure or integrating an etcd client in the application layer itself

For the full session notes see the detailed notes.

Action Items


 * Stop relying on a configuration repository to store the live state of clusters, it needs to be treated separately.
 * Use DNS for service location/discovery, and simple data structures, but has some limitations.
 * Use Confd with templates and scripts to manage more complex data structures.
 * For the MediaWiki specific case, there are a bunch of options:
 * generate a JSON file that is read by the application, parsed by MediaWiki and cached into APC.
 * generate a PHP file that will be included directly by the application.
 * Add safe measures to ensure that the configuration is consistent across the cluster and no hosts have a stale one.