Reading/Web/Projects/Edit based trending service

From MediaWiki.org
< Reading‎ | Web‎ | Projects
Jump to: navigation, search

The reading web team is building a trending service which responds to edit activity from our users, with the purpose of notifying users of interesting content.

This is an initiative to build upon the experiments of Pushipediaand Weekipediato process edit activity.

Use cases[edit]

The main immediate use case is push notifications, which will drive new traffic to newly created and relevant pages.

On the long term it is also hoped to be used to power lists of pages that have been relevant during a period of time, for example a week.

API[edit]

The API will expose the top pages that are receiving high edit activity. Each page will have a score. A negative or zero score means the page is not likely to be trending. The higher the score, the more likely a page is to be trending. When considering whether to display a page your client will want to consider the total number of edits and number of editors. The score is currently prone to changes and likely to fluctuate in size, so please do not rely too heavily on certain numbers meaning trending - instead use the score to get a relative sense of important compared to other things.

Working with the code[edit]

The API endpoint is provided by the mediawiki services api. Code lives in the mediawiki services trending repository. Bugs and feature requests are tracked in Phabricator.

Currently the service is only possible on the English Wikipedia due to the higher amount of edit traffic. Edit activity is subscribed to and processed. Although edits are the main driver of the trending algorithm, on the long term as APIs are made available, it will make use of other indicators such as page views.

Instructions for testing on vagrant[edit]

  1. Clone the repo inside your vagrant instance.
  2. Ensure Vagrant is setup with the eventbus role.
  3. SSH into the vagrant box and cd to the folder and install it. It's important to do this on Vagrant to avoid the need to setup kafka.
    vagrant ssh
    cd /vagrant/trending-edits/
    npm install
    
  4. Run `npm test` to verify the install worked correctly
  5. Configure `config.dev.yaml` like so:
    consume_dc: [ datacenter1 ]
    min_edits: 1
    
  6. Verify eventbus is on and working:
    sudo service eventlogging-service-eventbus status
    sudo service eventlogging-service-eventbus start
    
  7. In a new `vagrant ssh` window, download and execute the simulation script
    npm install node-fetch
    node reading-web-trending-simulator.js 
    wget -O - http://0.0.0.0:6927/en.wikipedia.org/v1/feed/trending-edits | tail
    
    Debugging: If you do not see a response with pages populated, please debug event bus.
  8. sudo tail -f /var/log/upstart/eventlogging-service-eventbus.log