Reading/Web/Projects/Edit based trending service
The reading web team built a trending service which responds to edit activity from our users, with the purpose of notifying users of interesting content.
It is currently not deployed to production but is available (unmaintained).
There is a similar service running on labs. A similar experimental service is maintained by Jon Robson in https://github.com/jdlrobson/wikipedia-trending-endpoint and https://wikipedia-trending.wmflabs.org
The main immediate use case was push notifications, which was hoped would drive new traffic to newly created and relevant pages.
On the long term it was also hoped to be used to power lists of pages that have been relevant during a period of time, for example a week.
The API exposes the top pages that are receiving high edit activity. Each page will have a score. A negative or zero score means the page is not likely to be trending. The higher the score, the more likely a page is to be trending. When considering whether to display a page your client will want to consider the total number of edits and number of editors. The score is currently prone to changes and likely to fluctuate in size, so please do not rely too heavily on certain numbers meaning trending - instead use the score to get a relative sense of important compared to other things.
Working with the code
Currently the service is only functional on wikis with high levels of traffic e.g. English Wikipedia due to the higher amount of edit traffic. Edit activity is subscribed to and processed. Although edits are the main driver of the trending algorithm, on the long term as APIs are made available, it will make use of other indicators such as page views.
Instructions for testing on vagrant
- Clone the repo inside your vagrant instance.
- Ensure Vagrant is setup with the eventbus role.
- SSH into the vagrant box and cd to the folder and install it. It's important to do this on Vagrant to avoid the need to setup kafka.
vagrant ssh cd /vagrant/trending-edits/ npm install
- Run `npm test` to verify the install worked correctly
- Configure `config.dev.yaml` like so:
consume_dc: [ datacenter1 ] min_edits: 1
- Verify eventbus is on and working:
sudo service eventlogging-service-eventbus status sudo service eventlogging-service-eventbus start
- In a new `vagrant ssh` window, download and execute the simulation scriptDebugging: If you do not see a response with pages populated, please debug event bus.
npm install node-fetch node reading-web-trending-simulator.js wget -O - http://0.0.0.0:6927/en.wikipedia.org/v1/feed/trending-edits | tail
sudo tail -f /var/log/upstart/eventlogging-service-eventbus.log