Notifications

'''This document is a work in progress. Comments are appreciated but this is not a final draft.''' '''This document requires clean up. Much of it is more technical than need be; technical parts should be excised to their own pages.'''

This document describes the design of a MediaWiki core feature, a notifications system, code-named Echo. This feature is designed to replace and augment existing notification systems as well as providing significantly more control to both users and developers as to how their notifications are handled, read, and deleted.

Nomenclature

 * Service - the central notification service.
 * Publisher - in this document, a publisher is a feature or extension that is a consumer of the publication/service API. Examples: LiquidThreads, Feedback Dashboard, Talk page edits
 * Event or Notification Event - A specific type of notification. Publishers must define and register notification events with the Service as well as the acceptable back-end pluggables they can use
 * Client - a single user viewing notifications from a single device. Users may have multiple devices.
 * Device - any system where the notification may be seen. Devices can include but are not limited to: user email, MediaWiki "in web" notifications, sms messages, etc.

Rationale
The central tracking bug, "MediaWiki needs a sane notification system", can be found at 32281.

There are several rationales for this feature, but primarily they fall into "providing a unified experience for the user" and "providing ease for developer implementation."

Currently, notifications within MediaWiki are handled poorly, primitively, and with a great deal of schizophrenic design. Multiple extensions and functions will fire off notifications and very few utilize the same system. This creates massive user confusion.

For example, the following systems utilize their own, distinct methods of user notifications:


 * Standard "user talk" and watchlist notifications
 * LiquidThreads replies and postings (Special:NewMessages)
 * Code Review reply notifications
 * Feedback Dashboard responses

On the user end of the issue, notifications are presented to the user is many different forms. A user may receive a talk page message in the form of an orange bar at the top of the page, or a LiquidThreads notifications in the form of a number in the top right corner, or a watchlist notification via email. Each different notification type is implemented differently and presents the event in a different manner, making the system confusing to users. A user should be able to easily maintain what notifications he/she wants to receive as well as be able to look at and act upon those notifications in an unambiguous manner. With the current spread-out and differentiated notifications system, this is impossible, especially in cases where there are multiple wikis on a farm and notification events occur on each individual wiki.

The notifications system also presents a development issue. Should the need arise to create a new type of notification or a new vector for delivering notifications, it would require numerous code changes in various modules of MediaWiki depending on what notification is being changed and how it is being changed. A centralized system would necessitate only one or two changes in code, usually in the form of an event registration hook or additional method in the notification system. This issue becomes amplified for extension developers because there is no infrastructure to work off of for sending notifications, so making a new type of notification would require either changes to the core, which causes issues when upgrading, or writing custom code, which only furthers the issue at hand.

Hypothesis
It is believed that having a well-designed and useful notifications system will:


 * Promote editor retention by:
 * Providing users with a modern experience that is similar to other sites
 * Providing a sense of activity within the various projects
 * Reducing the clutter on user talk pages
 * Providing an easier mechanism for displaying 360 interaction processes (as opposed to things like "talk back" templates), thus "centralizing" the discussion
 * Promote better software development by
 * Making it easier for extension developers to plug notifications into their code as needed
 * Making it easier for core developers to add new notification vectors
 * Providing a stable and platform-native interface pattern
 * Improve performance and efficiency by
 * Having all notifications stored in one location, thus requiring fewer database queries
 * Allowing wikis to push notifications to users at the system's convenience

Functional Requirements
Since this feature is comprehensive, requirements have been broken out into distinct sections. The "Full Monty" version of this feature requires significant back-end work involving multiple databases, and may be only of use to systems like Wikimedia Foundation wikis.

Cross-Wiki Notification
In order to facilitate easy handling of notifications in installations with multiple wikis, there must be a method for publishers in individual wikis to notify other wikis of local notifications to that the user can be informed. A comprehensive solution would involve creating a centralized notification server or designating one wiki as the central wiki. From there all notifications would be sent to the central server rather than kept and polled on each individual wiki.

Then, individual clients (a user on a specific wiki) will ask the server for all notifications on all wikis. Thus, one need only log into a single wiki (say, meta) and then be informed of activity they are interested in on the English Wikipedia, the German Wikipedia, MediaWiki.org, and any other connected system. Furthermore, links would be provided to the user to view and act upon these external notifications.

Notification Bundling
To avoid clutter, notifications from a single wiki of the same type may be "bundled" and collapsed.

For example, if four people edit a talk page, each edit will fire a notification. However, with bundling, instead of the user being shown four notifications, thus:


 * Brion edited your talk page
 * Jorm edited your talk page
 * Eloquence edited your talk page
 * Kaldari edited your talk page

They would instead be shown something like this:


 * Brion, Jorm, Eloquence and 1 other edited your talk page

or even this:


 * 4 users edited your talk page

There are a number of interesting variations of how these things can be displayed.

Event Preferences
In the Echo notification system, there exists a variety of event types and a variety of event vectors. Furthermore, each event type may be delivered to the user through any number of the possible vectors. Which notifications a user receives and through which vectors he/she receives them must be a user preference that can be customized.

Users will be given the ability to decide which events they receive notifications for and through which means they receive them. For example, a user may wish to get Notifications when someone edits their talk page and when someone responds to a Feedback Dashboard comment, but not when a page on their watchlist is edited, but when someone reverts their edits. Further, users may choose to receive talk page notifications over both phone and email, but only receive edit reversions through in-wiki notification.

Pluggable Backends
This section concerns itself with Notification delivery - where do Notifications appear and how can they be retrieved?

Email
Some things can trigger email notifications, including the things that will trigger user-talk notifications today; but also other events may trigger emails such as Extension:CodeReview's followup notifications.

These should be brought into the common notification system so they're available over non-email channels, and so that the email channels include relevant information (such as the content of a message!)


 * 32285 Email backend for notifications to supplement/consolidate enotif with common notification infrastructure

Web UI
The existing new messages links in the dynamically-generated web UI of MediaWiki will need to be updated to read from the Central Service.

API
An API interface to poll for new messages would be useful for extending the front-end web UI so that you still see notifications even if you're on a page for a long time (say you're editing long text, or using an AJAX-y interface for page patrolling). This would also be a first step for mobile apps and other tools to hook into the same interface.

(Possible API? Notification triggers a remote HTTP POST?)

Further enhancement could be done with WebSockets or long-polling techniques, but may require server-side support for long-running connections.


 * 32283 API point for polling notification data
 * 32284 Show live talk & other notification updates in web UI
 * 32292 Show watchlist & talk notifications in MobileFrontend web UI

Mobile push notifications
Various mobile operating systems like Apple's iOS and Google's Android have message proxying systems that allow an application to register to efficiently listen for notifications sent through an intermediary service, even when the application has been closed to save memory & power.

Backends to set up the device subscriptions and send pings through the proxy will be a benefit here.


 * 32288 Android cloud-to-device messaging server-side
 * 32289 Android app support
 * 32287 iOS push notifications server-side
 * 32291 iOS app support

XMPP
Notification systems could be pushed onto users via instant message on various networks (i.e. Jabber). Similar thing could be echoed on proprietary messaging networks, and by bots on channels on chat networks since the service mechanisms (like eJabberd) have pluggable features for relaying messages to transport gateways.

Note that XMPP is of interest also for XMPP recent changes, so common infrastructure for "firehose" and smaller notifications may be useful.

Existing Notifications
The general approach will be to first reach feature parity with the existing notifications systems. Here is a list of the currently known notifications. The project team will need to evaluate these notifications to see which ones should be incorporated into the scope of Echo:


 * Email when (assuming email is provided):
 * Page on my watchlist has changed
 * My user talk page is changed
 * Someone writes me an email through the Email this user feature
 * email another user through the Email this user feature
 * When a page is created (e.g. if you watch Foo, you also get an e-mail when Talk:Foo is created).
 * You get a "my user page" e-mail if you watch it (with a very confusing edit summary talking about creation) when a manual user rights change happens (+sysop or +rollbacker etc.).


 * Show Yellow bar when:
 * My user talk page has changed


 * Other notifications
 * CentralNotice, sitenotice, geonotices, etc. as describe on here.

Infrastructure
As stated in the rationale, this new notifications system must also address ease-of-development issues for both core and extensions developers when changing or adding event types or vectors.

The new notification system must provide an easy method, using some sort of hook or method, to register new message types. Extension developers then may simply register their type with the notifications system, and MediaWiki will automatically set up user preferences for the notification and automatically deliver those notifications as required. The extension itself will only be responsible for actually triggering and publishing the notification.

The system must also provide a method of registering new vectors, or methods of notifying the user of an event. Existing examples are email and in-wiki UI notifications, but an extension developer may want to implement a custom vector, e.g., sending out notifications to an external service for integration purposes.

Entities

 * Event
 * Echo revolves around events. An Event is something that occurs that may result in one or more notifications. For example, "User X edited the talk page for user Y", "User X edited User talk:Y" and "User X reverted user Z's edit" are all events. They may in fact refer to the same underlying action, but for the purposes of Echo they are considered separate events. The most important attribute of an Event is its "type", which determines its handling within Echo.


 * Action
 * An Action is something that is triggered when an Event occurs. A Subscription rule will determine which Actions are fired.


 * Notification
 * A Notification associates a Consumer with an Event. Echo dispatches Notifications when an event occurs that causes the 'notify' Action to be taken for a particular Consumer. Notifications are transient, and stored for a fixed period of time. They can be retrieved via the API or via the Web Interface.


 * Subscription
 * A Subscription is a statement that a particular Consumer is interested in a particular class of events. A subscription specifies an event type and optionally a page. Any event with that type (and on that page, if a page is specified) will result in a determined set of actions occurring. These actions may be explicitly specified or implicit for that type of event.
 * Subscriptions may be explicit or implicit. Echo maintains a list of explicit (Consumer X wants to be notified and emailed when event Y occurs on page Z) and implicit (All Consumers should be notified and emailed when the event 'edit-talk-page' occurs on their talk page) Subscription rules. Implicit rules can be overridden by explicit rules.


 * Consumer
 * A Consumer is somebody who cares about Events. Currently the only type of Consumer is a MediaWiki user account.

Echo Core
Echo Core is the Echo system proper. It accepts Events from publishers, maintains explicit and implicit Subscription rules, and materialises events. An Event is materialised by resolving the Subscription rules that apply to it and firing the Actions stipulated by those rules. Materialisation will not always happen immediately – the time when Events are materialised is determined by the Queueing System.

Event Publishers
An Event Publisher is a part of MediaWiki that creates Events and submits them to Echo. Once an Event has been submitted to Echo, the obligation of the Publisher has ended.

Publishers will often also add their own implicit Subscription rules and Notification Formatters.

Notification Formatters
A Notification Formatter accepts a configuration (which is typically set on a per-event-type basis), and a Notification (with its corresponding Consumer and Event) and spits out a representation of the Notification. It may support a number of output formats, including HTML, Text and Email (subject and body).

Queueing System
The queueing system determines when Events are materialised. This will depend on the required latency (higher for email, lower for notifications to infrequent visitors), the publish rate of the Publisher, the number and consumption characteristics of the consumers, and other undetermined factors.

Currently all events are materialised in order from the Job Queue.

User Notification Preferences
With a potentially limitless number of clients, it is not feasible for users to be expected to adjust their preferences on a per-wiki basis. As such, each wiki will have a notification preferences page that reads in a list of all Publishers and the Event types from the Central Service.

The user can then indicate which Events they wish to be notified about and which back-ends they wish to receive the notifications on. These preferences are then stored on the Central Service, not the local wiki.

User Experience
This section describes the user experience of the notifications system.

Note: It is important to note that all notifications exist in a state of "read" or "not read". The process by which a notification is marked as "read" is something that we will want to experiment with. There are several ways of handling this:

Notifications can be marked as "read" when:
 * 1) They have been "seen" (e.g., when the user views their notifications)
 * 2) They have been "acted upon" (e.g., the user visits their talk page after being notified it was edited)
 * 3) They are manually marked read

The mockup provided assumes that notifications are marked read upon being seen.

Elements and Process
A new user interface link, "Notifications," will appear in the user's chrome. If the user has "unread" notifications, the number of unread notifications will appear next to the link as a badge upon a red background. This badge will auto-update (see below).

Clicking on the "Notifications" link or the badge will display the Notifications Bar (clicking on it again, or anywhere outside of the Notifications Bar, will close the bar, possibly marking all Notifications as read).

Notifications are "bucketed" by the wiki they have come from. Individual wiki buckets will contain all unread notifications from that wiki, regardless of timestamp.

Buckets are sorted top-to-bottom based on the most recent timestamp held within the bucket. Thus, it is possible for a bucket at the top of the bar to contain Notifications that are older than those contained in buckets beneath it provided that there are Notifications that are newer than the most recent Notification in lower buckets.

Buckets will have a header that indicates which wiki they are coming from (both its "official" name and the "URL"). Headers are not hot.

Inside each bucket, Events are displayed. Events may be atomic or bundled, depending on the preference of the Publisher. Events contain timestamps printed in context-relative format (e.g., "10 minutes ago, "3 hours ago"), with the most recent listed first.

Event text should really be capped at 250 characters.

Events should have distinguishable icons. (Should the process of Event-to-Icon association should happen at the Central Service layer or at the Publisher layer? Or skin?)

Clicking on any one Event's entry will cause that Event to be "actioned" - usually going to view the item in question.

The entire section of the Event is "hot". While some things (such as user names) may appear to be hot links, they are not - the entire element is and is focused on a single URL.

At the bottom of the Notifications Bar will be an additional link, "View all notifications". Clicking this will bring the user to a Special page which lists all notifications, read and unread.

Dynamic Updates
The notifications "bar" will periodically poll the Central Service for new Notifications to display. The Central Service will return a JSON object which tells the Client that new Notifications have arrived.

Updates will change the number of unread notifications in the badge.

Open issue: if the full list is "open", will this suspend polling? This may be desirable as elements may move around on the bar.

Deleted Elements
The following things will be going away forever, being replaced by the Echo system:
 * the "yellow bar" notifications about talk page edits,
 * LiquidThreads "New Messages" link and special page,
 * others as determined.

Use Cases
The following is a list of use cases taken from the RFC talk page. This list shouldn't be taken too literally. The cases here are meant to provide an idea of the types of emails that the notification should support.

These also assume the lack of a threaded discussion system.

(not in any specific order)


 * When someone leaves a message on my talk page, I receive an email which includes the following information:
 * Username of the person who wrote on my talk page
 * Content of the message left on my talk page (including section heading)
 * Instructions on how to reply
 * When I register for an account on Wikipedia, I get an email welcoming me.
 * When I complete my first successful edit on Wikipedia, I get an email congratulating me. Email contains more ways for me to get involved.
 * When I reach autoconfirmed status, I get an email letting me know I can edit BLPs’s (I’m not suggesting we actually do this -- I’m just trying to give a sense for the type of functionality).
 * When I complete 100/500/1000/etc edits, I receive an email congratulating me.
 * After completing 10 edits and xx amount of time, I receive an email with “tutorial” type information. (Idea here is that we send out more tutorial information on syntax, policy, etc. for editors who show that they are well-intentioned).
 * When someone reverts my edit, I receive an email letting me know what happened and why.
 * When I submit a page for creation, I receive an email letting me know that my page has been submitted for creation.
 * When the page I submitted changes state, I receive an email letting me know what happened.
 * I receive an email when someone expands an article I created, letting me know what the additions were and who made them.
 * When I rate my first article, I receive an email thanking me. (Rating could mean adding a comment)
 * When my rating comment gets promoted to the talk page, I get a message letting me know, with a link to the talk page discussion.
 * When I rate my 5th article, I receive an email thanking me for my contributions, and encouraging me to try to edit.
 * I can opt to receive periodic emails about Wikip/media developments (e.g., new features)
 * I can sign up for emails about specific Wikiprojects.
 * If I stop editing for x weeks, I receive an email encouraging me to resume editing.
 * When a page I recently edited gets a template, I receive an email letting me know what happened in plain language.
 * When my talk page gets a template, I receive an email letting me know what happened in plain language.
 * When a bot does something, I receive an email letting me know what the bot did, without referring to the bot/requiring me to know what a bot is.


 * Email controls:
 * I can opt to receive no emails.
 * I can opt to receive all my emails in a daily or weekly digest.
 * I can consolidate emails by type (e.g., all emails about users editing my talk page)
 * I can opt out of emails by type (e.g., automatic congrats emails, talk page notifications, etc.)


 * We should have ways to ask for emails from users that haven't provided (e.g., user submits a MoodBar comment but hasn't provided an email address).

Additional documents

 * Extension page including installation instructions