Reading/Web/Projects/Edit based trending service
The reading web team is building a trending service which responds to edit activity from our users, with the purpose of notifying users of interesting content.
The main immediate use case is push notifications, which will drive new traffic to newly created and relevant pages.
On the long term it is also hoped to be used to power lists of pages that have been relevant during a period of time, for example a week.
The API will expose the top pages that are receiving high edit activity. Each page will have a score. A negative or zero score means the page is not likely to be trending. The higher the score, the more likely a page is to be trending. When considering whether to display a page your client will want to consider the total number of edits and number of editors. The score is currently prone to changes and likely to fluctuate in size, so please do not rely too heavily on certain numbers meaning trending - instead use the score to get a relative sense of important compared to other things.
Working with the code
Currently the service is only possible on the English Wikipedia due to the higher amount of edit traffic. Edit activity is subscribed to and processed. Although edits are the main driver of the trending algorithm, on the long term as APIs are made available, it will make use of other indicators such as page views.
Instructions for testing on vagrant
- Clone the repo inside your vagrant instance.
- Ensure Vagrant is setup with the eventbus role.
- SSH into the vagrant box and cd to the folder and install it. It's important to do this on Vagrant to avoid the need to setup kafka.
vagrant ssh cd /vagrant/trending-edits/ npm install
- Run `npm test` to verify the install worked correctly
- Configure `config.dev.yaml` like so:
consume_dc: [ datacenter1 ] min_edits: 1
- Verify eventbus is on and working:
sudo service eventlogging-service-eventbus status sudo service eventlogging-service-eventbus start
- In a new `vagrant ssh` window, download and execute the simulation scriptDebugging: If you do not see a response with pages populated, please debug event bus.
npm install node-fetch node reading-web-trending-simulator.js wget -O - http://0.0.0.0:6927/en.wikipedia.org/v1/feed/trending-edits | tail
sudo tail -f /var/log/upstart/eventlogging-service-eventbus.log