Reading/Web/Projects/Edit based trending service

The reading web team built a trending service which responds to edit activity from our users, with the purpose of notifying users of interesting content.

This was an initiative to build upon the experiments of Pushipediaand Weekipediato process edit activity.

It is currently not deployed to production but is available (unmaintained).

There is a similar service running on labs. A similar experimental service is maintained by Jon Robson in https://github.com/jdlrobson/wikipedia-trending-endpoint and https://wikipedia-trending.wmflabs.org

Use cases

The main immediate use case was push notifications, which was hoped would drive new traffic to newly created and relevant pages.

On the long term it was also hoped to be used to power lists of pages that have been relevant during a period of time, for example a week.

API

The API exposes the top pages that are receiving high edit activity. Each page will have a score. A negative or zero score means the page is not likely to be trending. The higher the score, the more likely a page is to be trending. When considering whether to display a page your client will want to consider the total number of edits and number of editors. The score is currently prone to changes and likely to fluctuate in size, so please do not rely too heavily on certain numbers meaning trending - instead use the score to get a relative sense of important compared to other things.

Working with the code

The API endpoint is provided by the mediawiki services api. Code lives in the mediawiki services trending edits repository. Bugs and feature requests are tracked in Phabricator.

Currently the service is only functional on wikis with high levels of traffic e.g. English Wikipedia due to the higher amount of edit traffic. Edit activity is subscribed to and processed. Although edits are the main driver of the trending algorithm, on the long term as APIs are made available, it will make use of other indicators such as page views.

Instructions for testing on vagrant

1. Clone the repo inside your vagrant instance.
2. Ensure Vagrant is setup with the eventbus role.
3. SSH into the vagrant box and cd to the folder and install it. It's important to do this on Vagrant to avoid the need to setup kafka.

   vagrant ssh
   cd /vagrant/trending-edits/
   npm install
   

4. Run `npm test` to verify the install worked correctly
5. Configure `config.dev.yaml` like so:

   consume_dc: [ datacenter1 ]
   min_edits: 1
   

6. Verify eventbus is on and working:

   sudo service eventlogging-service-eventbus status
   sudo service eventlogging-service-eventbus start
   

7. In a new `vagrant ssh` window, download and execute the simulation script

   npm install node-fetch
   node reading-web-trending-simulator.js 
   wget -O - http://0.0.0.0:6927/en.wikipedia.org/v1/feed/trending-edits | tail
   

   Debugging: If you do not see a response with pages populated, please debug event bus.

8. sudo tail -f /var/log/upstart/eventlogging-service-eventbus.log