User:AGreen (WMF)/Draft:Dev setup for centralnotice changes monitor

Overview
checks the CentralNotice API and consumes Kafka messages to follow changes in banners and transcluded pages. Main requirements are: Python 3, MariaDB, a MediaWiki installation with the CentralNotice and EventBus extensions installed, and Kafka.

The way I've achieved this setup is to start with Vagrant on the eventbus role enabled, and install CentralNotice manually there. That role provides a Kafka setup similar to WMF production. I've then tweaked Vagrant to allow consumption of Kafka streams on the host machine, which is where I've run  itself. Other setups, including running the script from within Vagrant, are possible, too.

eventbus role
Enable the eventbus role and provision.

Notes:


 * 1) This didn't work for me with the fundraising role enabled at the same time. Probably no harm in trying it, though, if anyone would prefer not to disable that role.
 * 2) When I first enabled this role, a minor bug caused provisioning to fail, and I had to tweak a file to fix it. I don't remember which file it was. The bug may be fixed by now. If provisioning does fail for you, please ping me. IIRC the fix was easy to find based on the error message.

CentralNotice
Note: If you already have CentralNotice installed in Vagrant in the main MediaWiki instance, skip this step.

Clone CentralNotice in.

Add the following lines to. (Note: There might be a better place to put this.) In the CentralNotice directory, from within the Vagrant box, run. Then, also from within the Vagrant box, from, run.

EventBus service
From inside the Vagrant box, check that the  service is running:

If an error is shown, re-start the service:

Wait 5-10 seconds, then check the status again. Note: In my current set up, I have to do this every time I reboot the Vagrant box. No idea why.

Consuming Kafka streams from the host machine
To run on the host machine,  needs to consume Kafka streams produced from the Vagrant box. (If you'll run the script from within Vagrant, this step is not necessary.)

Copy  to   and add the following lines: Then re-start the Vagrant box. (It might be necessary to re-provision, too—I'm not sure.)

On the host machine, add the following to :

Checking Kafka streams
You can use kafkacat to check that Kafka streams are being produced.

You should see an event for any edit to any page. Here is the list of other topics you can connect to.

Note: If this command fails, just try it again!

Database
Create a database and a database user, grant the user rights on the database, then run the following command (substituting database, user and password as appropriate):

For development purposes, the SQL to drop all tables is also provided. To use it, copy   as   and uncomment the last two lines.

Then, you can reset the database like this:

(Do not deploy an uncommented version of the drop tables file to production. Using the filename  for the uncommented version will prevent it from being added to the Git repository.)

Installation
Clone from the temporary GitHub repo.

Run the following command from the repository root directory to install with dependencies. (This will create a symlink from a normal python library location to wherever you actually put the repo.)

Configuration
Copy  to. Replace database settings with whatever you used when you created the database and database user.

Execution
From the repository root directory, run:

Optionally you can add the  flag, or use   to point to a different configuration file.

Every change that comes down the pipe should modify the contents of the  table in the database, so keeping an eye on that can be useful for debugging.

Possible issues
It seems that sometimes editing banners via the banner editor interface does not trigger a recent change event and does not change core's listings about what is transcluded where. Note that it might be that the event/changes are just not triggered right away. This may be a core or a CentralNotice issue... For the purposes of testing the script, I'd recommend modifying banners as wiki pages, using the "Edit banner on-wiki" link at the top of the CentralNotice banner editor.