Requests for comment/Parallel maintenance scripts

See additional details in.

Proposal

 * Add MediaWiki\Parallel namespace with helper classes, living in includes/parallel
 * ParallelMaintenance class wraps helpers to add  option to farm sub-tasks to multiple workers
 * port various Maintenance classes to ParallelMaintenance which can make use of it

Implementation

 * core change adding ParallelMaintenance and MediaWiki\Parallel\* helpers and porting several maint scripts: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/451099
 * TimedMediaHandler change using ParallelMaintenance in a test script: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TimedMediaHandler/+/232214

Details
In MediaWiki\Parallel are some interfaces:
 * IController -- implemented by the StreamController classes; allows enqueueing items into the work stream and syncing the state
 * IConsumer --  performs a work item with serializable data input and passes data back out
 * IDispatcher --  is where implementors provide input data and dispatch it into   for work by an IConsumer later...   accepts the return value from the consumer.

Concrete implementations:
 * InProcessController - just passes queued items directly to a consumer for immediate execution
 * StreamController - base class running JSON-serialized data to a number of child threads over i/o streams
 * ForkStreamController - pcntl_fork-based implementation, takes an IDispatcher and an IConsumer
 * ExecStreamController - proc_open-based implementation, takes an IDispatcher and a command line; caller's responsibility to route the command line to something that runs a StreamWorker
 * StreamWorker - child process side for StreamController. Takes an IConsumer.

\ParallelMaintenance extends \Maintenance and implements the IDispatcher and IConsumer interfaces with abstract process, loop, and result methods to be implemented by the final user class. Several maintenance scripts have been ported to use this, and it seems to work ok.

Open questions

 * Increased use of pcntl_fork-based maintenance scripts might be fragile; though they close connections it's likely that some new or extension feature might get forgotten; weird teardown issues are also visible if child processes aren't carefully killed in unusual contexts like unit testing. Would it be better to move ParallelMaintenance to use the proc-open-based controller?
 * Considerations on error handling in worker contexts?
 * Should ForkController be fully merged with MediaWiki\Parallel?
 * Should these IDispatch, IConsumer interfaces be used or just pass callables around? If interfaces are used, are the method names too generic?
 * IDispatch::loop method could be a generator too, using yield instead of an IController::query callback. Good/bad/indifferent idea?
 * Considerations on possible web-server-side use of parallelism where pcntl_fork isn't available and proc_open could be expensive?