User:AGreen (WMF)/Draft:Dev setup for centralnotice changes monitor

Overview
checks the CentralNotice API and consumes Kafka messages to follow changes in banners and transcluded pages. Main requirements are: Python 3, MariaDB, a MediaWiki installation with the CentralNotice and EventBus extensions installed, and Kafka.

The way I've achieved this setup is to start with Vagrant on the eventbus role enabled, and install CentralNotice manually there. That role provides a Kafka setup similar to WMF production. I've then tweaked Vagrant to allow consumption of Kafka streams on the host machine, which is where I've run  itself. Other setups, including running the script from within Vagrant, are possible, too.

eventbus role
Enable the eventbus role and provision.

Notes:


 * 1) This didn't work for me with the fundraising role enabled at the same time. Probably no harm in trying it, though, if anyone would prefer not to disable that role.
 * 2) When I first enabled this role, a minor bug caused provisioning to fail, and I had to tweak a file to fix it. I don't remember which file it was. The bug may be fixed by now. If provisioning does fail for you, please ping me. IIRC the fix was easy to find based on the error message.

CentralNotice
Note: If you already have CentralNotice installed in Vagrant in the main MediaWiki instance, skip this step.

Clone CentralNotice in.

Add the following lines to. (Note: There might be a better place to put this.) In the CentralNotice directory, from within the Vagrant box, run. Then, also from within the Vagrant box, from, run.

EventBus service
From inside the Vagrant box, check that the  service is running:

If an error is shown, re-start the service:

Wait 5-10 seconds, then check the status again. Note: In my current set up, I have to do this every time I reboot the Vagrant box. No idea why.

Consuming Kafka streams from the host machine
To run on the host machine,  needs to consume Kafka streams produced from the Vagrant box. (If you'll run the script from within Vagrant, this step is not necessary.)

Copy  to   and add the following lines: Then re-start the Vagrant box. (It might be necessary to re-provision, too—I'm not sure.)

On the host machine, add the following to :

Checking Kafka streams
You can use kafkacat to check that Kafka streams are being produced.

You should see an event for any edit to any page. Here is the list of other topics you can connect to.

Note: If this command fails, just try it again!

Clone from the temporary GitHub repo.

Run the following command from the repository root directory to install with dependencies. (This will create a symlink from a normal python library location to wherever you actually put the repo.)

Database
Create a database and a database user, grant the user rights on the database, then run the following command (substituting database, user and password as appropriate):

For development purposes, the SQL to drop all tables is also provided. To use it, copy   as   and uncomment the last two lines.

Then, you can reset the database like this:

(Do not deploy an uncommented version of the drop tables file to production. Using the filename  for the uncommented version will prevent it from being added to the Git repository.)

Copy  to. Replace database settings with whatever you used when you created the database and database user.