Echo (Notifications)

WMF engineering activity

Notifications

Initial ideas for the Notifications infrastructure and UI

Group:	Features
Start:	2012-07-01
End:
Management:	Brion Vibber, Terry Chay
Team:	Brandon Harris, WMF team TBD, (Wikia)

This document is a work in progress. Comments are appreciated but this is not a final draft.

The Echo notifications system, showing by-wiki filtering and event rollups.

This document requires clean up. Much of it is more technical than need be; technical parts should be excised to their own pages.

This document describes the design of a MediaWiki core feature, a notifications system, code-named Echo. This feature is designed to replace and augment existing notification systems as well as providing significantly more control to both users and developers as to how their notifications are handled, read, and deleted.

Nomenclature

• Service - the central notification service.
• Publisher - in this document, a publisher is a feature or extension that is a consumer of the publication/service API. Examples: LiquidThreads, Feedback Dashboard, Talk page edits
• Notification Event - A specific type of notification. Publishers must define and register notification events with the Service as well as the acceptable back-end pluggables they can use
• Client - a single user viewing notifications from a single device. Users may have multiple devices.
• Device - any system where the notification may be seen. Devices can include but are not limited to: user email, MediaWiki "in web" notifications, sms messages, etc.

Rationale

The central tracking bug, "MediaWiki needs a sane notification system", can be found at bugzilla:32281.

There are several rationales for this feature, but primarily they fall into "providing a unified experience for the user" and "providing ease for developer implementation."

Notification Source Unification

Currently, notifications within MediaWiki are handled poorly, primatively, and with a great deal of schizophrenic design. Multiple extensions and functions will fire off notifications and very few utilize the same system. This creates massive user confusion.

For example, the following systems utilize their own, distinct methods of user notifications:

• Standard "user talk" and watchlist notifications
• LiquidThreads replies and postings (Special:NewMessages)
• Code Review reply notifications
• Feedback Dashboard responses

This not an exhaustive list. Planning for future functionality reveals possibly hundreds of additional notification vectors. Quite simply, the current model is unsustainable.

Hypothesis

It is believed that having a well-designed and useful notifications system will:

• Promote editor retention by:
  • Providing users with a modern experience that is similar to other experiences
  • Providing a sense of activity within the various projects
  • Reducing the clutter on user talk pages
  • Providing an easier mechanism for displaying 360 interaction processes (as oppposed to things like "talk back" templates), thus "centralizing" the discussion
• Promote better software development by
  • Making it easier for extension developers to plug notifications into their code as needed
  • Providing a stable and platform native interface pattern

Functional Requirements

Since this feature is comprehensive, requirements have been broken out into distinct sections. The "Full Monty" version of this feature requires significant back-end work involving multiple databases, and may be only of use to systems like Wikimedia Foundation wikis.

Cross-Wiki Notification

A truly comprehensive solution will broadcast notifications from all wikis to a central "notification server". Then, individual clients (a user on a specific wiki) will ask the server for all notifications on all wikis.

Thus, one need only log into a single wiki (say, meta) and then be informed of activity they are interested in on the English Wikipedia, the German Wikipedia, MediaWiki.org, and any other connected system.

Notification Bundling

To avoid clutter, notifications from a single wiki of the same type may be "bundled" and collapsed.

For example, if four people edit a talk page, each edit will fire a notification. However, with bundling, instead of the user being shown four notifications, thus:

• Brion edited your talk page
• Jorm edited your talk page
• Eloquence edited your talk page
• Kaldari edited your talk page

They would instead be shown something like this:

• Brion, Jorm, Eloquence and 1 other edited your talk page

or even this:

• 4 users edited your talk page

There are a number of interesting variations of how these things can be displayed.

Event Filtering

Users will require the ability to "filter" which Events they receive Notifications for. For example, a user may wish to get Notifications when someone edits their talk page and when someone responds to a Feedback Dashboard comment, but not when a page on their watchlist is edited, but when someone reverts their edits.

Further, users may choose to say "I wish notifications of this type to be sent to my phone, but not email."

Further, users may choose to ignore entire classes of Events.

Pluggable Backends

This section concerns itself with Notification delivery - where do Notifications appear and how can they be retrieved?

Email

Some things can trigger email notifications, including the things that will trigger user-talk notifications today; but also other events may trigger emails such as Extension:CodeReview's followup notifications.

These should be brought into the common notification system so they're available over non-email channels, and so that the email channels include relevant information (such as the content of a message!)

• bugzilla:32285 Email backend for notifications to supplement/consolidate enotif with common notification infrastructure

Web UI

The existing new messages links in the dynamically-generated web UI of MediaWiki will need to be updated to read from the Central Service.

API

An API interface to poll for new messages would be useful for extending the front-end web UI so that you still see notifications even if you're on a page for a long time (say you're editing long text, or using an AJAX-y interface for page patrolling). This would also be a first step for mobile apps and other tools to hook into the same interface.

(Possible API? Notification triggers a remote HTTP POST?)

Further enhancement could be done with WebSockets or long-polling techniques, but may require server-side support for long-running connections.

• bugzilla:32283 API point for polling notification data
  • bugzilla:32284 Show live talk & other notification updates in web UI
  • bugzilla:32292 Show watchlist & talk notifications in MobileFrontend web UI

Mobile push notifications

Various mobile operating systems like Apple's iOS and Google's Android have message proxying systems that allow an application to register to efficiently listen for notifications sent through an intermediary service, even when the application has been closed to save memory & power.

Backends to set up the device subscriptions and send pings through the proxy will be a benefit here.

• bugzilla:32288 Android cloud-to-device messaging server-side
  • bugzilla:32289 Android app support
• bugzilla:32287 iOS push notifications server-side
  • bugzilla:32291 iOS app support

XMPP

Notification systems could be pushed onto users via instant message on various networks (i.e. Jabber). Similar thing could be echoed on proprietary messaging networks, and by bots on channels on chat networks since the service mechanisms (like eJabberd) have pluggable features for relaying messages to transport gateways.

Note that XMPP is of interest also for XMPP recent changes, so common infrastructure for "firehose" and smaller notifications may be useful.

Technical Architecture

The Echo architecture is a three-pronged model consisting of a Central Service, multiple Publishers, and multiple user Clients.

System Roles

Central Service

The central service is the master notification server. The Central Service has four primary duties:

• Accept registrations from Publishers and know what to do with them.
• Accept Events from Publishers and know what to do with them
• Accept "notification pull" requests from Clients and know what to do with them.
• Store user preferences regarding notifications (see below)

Publisher

A publisher is a feature or extension enabled on any wiki. Publishers know what kinds of Events they accept and know how to register these events with the Central Service.

Publishers also know how and when to fire Events to the Central Service. Events are fired by either user action or automated activity.

Publishers on local wikis are expected to hold the actual "text" of the notification locally. Spamming this data to the Central Service will be overwhelming. Instead, the Central Service needs only know that there is a notification.

Clients

Clients receive notifications from the Central Service. In most cases, like through a web interface, this will actually be handled via "pull requests" through Javascript or page loads (if Javascript is disabled).

Registration Process

1. Upon startup, each wiki that has Publishers (including the core MediaWiki software itself) connects to the Central Service and registers its Event types and possible handlers with the Service.
2. The Central Service returns acknowledgement (READY) or error to the Publisher.

For example, upon startup, LiquidThreads may connect to the Central Service and register the following events with arguments (pseudocode):

1. NEW_THREAD_TO_TALK_PAGE(pageID, authorID, targetUser, date, notificationID, bundleable=no);
2. NEW_REPLY_TO_THREAD(threadID, authorID, targetUser, date, notificationID, bundleable=yes);
3. THREAD_MOVE(threadID, pageID, authorID, targetUser, date, notificationID, bundleable=no)

In this case, "targetUser" = the user who is to receive the notification, and "notificationID" is the id for the text kept locally.

Obviously, some systems may have extremely high volume (edits to highly watched pages, for instance). It may be that "targetUser" is an array, or there may be other, more efficient ways to handle this case.

Event Notification Process

1. An Event will occur within a system that a Publisher watches. Events can be triggered by user action or by automated processes.
   1. The Publisher packages a Notification for the Event. It stores human-readable text in a local table on the wiki. It then broadcasts the Notification to the Central Service.
2. A user client connects (periodically) to the Central Service and issues a pull request for all Notifications pertaining to the user.
   1. The Central Service loads all Notifications to be sent to the user from its own database.
   2. The Central Service contacts all satellite wikis that it needs to in order to retrieve the text of the notifications.
   3. The Central Service handles "bundling" requests (should this be done here?)
   4. The Central Service returns the Notifications to the client
3. The Client acts upon the Notifications, possibly marking them as read.
4. When a Notification is marked as read, this flag is sent back to the Central Service.

User Notification Preferences

With a potentially limitless number of clients, it is not feasible for users to be expected to adjust their preferences on a per-wiki basis. As such, each wiki will have a notification preferences page that reads in a list of all Publishers and the Event types from the Central Service.

The user can then indicate which Events they wish to be notified about and which back-ends they wish to recieve the notifications on. These preferences are then stored on the Central Service, not the local wiki.

User Experience

This section describes the user expererience of the notifications system.

Note: It is important to note that all notifications exist in a state of "read" or "not read". The process by which a notification is marked as "read" is something that we will want to experiment with. There are several ways of handling this:

Notifications can be marked as "read" when:

1. They have been "seen" (e.g., when the user views their notifications)
2. They have been "acted upon" (e.g., the user visits their talk page after being notified it was edited)
3. They are manually marked read

The mockup provided assumes that notifications are marked read upon being seen.

Elements and Process

A new user interface link, "Notifications," will appear in the user's chrome. If the user has "unread" notifications, the number of unread notifications will appear next to the link as a badge upon a red background. This badge will auto-update (see below).

Clicking on the "Notifications" link or the badge will display the Notifications Bar (clicking on it again, or anywhere outside of the Notifications Bar, will close the bar, possibly marking all Notifications as read).

Notifications are "bucketed" by the wiki they have come from. Individual wiki buckets will contain all unread notifications from that wiki, regardless of timestamp.

Buckets are sorted top-to-bottom based on the most recent timestamp held within the bucket. Thus, it is possible for a bucket at the top of the bar to contain Notifications that are older than those contained in buckets beneath it provided that there are Notifications that are newer than the most recent Notification in lower buckets.

Buckets will have a header that indicates which wiki they are coming from (both its "official" name and the "url"). Headers are not hot.

Inside each bucket, Events are displayed. Events may be atomic or bundled, depending on the preference of the Publisher. Events contain timestamps printed in context-relative format (e.g., "10 minutes ago, "3 hours ago"), with the most recent listed first.

Event text should really be capped at 250 characters.

Events should have distinguishable icons. (Should the process of Event-to-Icon association should happen at the Central Service layer or at the Publisher layer? Or skin?)

Clicking on any one Event's entry will cause that Event to be "actioned" - usually going to view the item in question.

The entire section of the Event is "hot". While some things (such as user names) may appear to be hot links, they are not - the entire element is and is focused on a single url.

At the bottom of the Notifications Bar will be an additional link, "View all notifications". Clicking this will bring the user to a Special page which lists all notifications, read and unread.

Dynamic Updates

The notifications "bar" will periodically poll the Central Service for new Notifications to display. The Central Service will return a JSON object which tells the Client that new Notifications have arrived.

Updates will change the number of unread notifications in the badge.

Open issue: if the full list is "open", will this suspend polling? This may be desirable as elements may move around on the bar.

Deleted Elements

The following things will be going away forever, being replaced by the Echo system:

• the "yellow bar" notifications about talk page edits,
• LiquidThreads "New Messages" link and special page,
• others as determined.

Use Cases

The following is a list of use cases taken from the RFC talk page. This list shouldn't be taken too literally. The cases here are meant to provide an idea of the types of emails that the notification should support.

These also assume the lack of a threaded discussion system.

(not in any specific order)

• When someone leaves a message on my talk page, I receive an email which includes the following information:
  • Username of the person who wrote on my talk page
  • Content of the message left on my talk page (including section heading)
  • Instructions on how to reply
• When I register for an account on Wikipedia, I get an email welcoming me.
• When I complete my first successful edit on Wikipedia, I get an email congratulating me. Email contains more ways for me to get involved.
• When I reach autoconfirmed status, I get an email letting me know I can edit BLPs’s (I’m not suggesting we actually do this -- I’m just trying to give a sense for the type of functionality).
• When I complete 100/500/1000/etc edits, I receive an email congratulating me.
• After completing 10 edits and xx amount of time, I receive an email with “tutorial” type information. (Idea here is that we send out more tutuorial information on syntax, policy, etc. for editors who show that they are well-intentioned).
• When someone reverts my edit, I receive an email letting me know what happened and why.
• When I submit a page for creation, I receive an email letting me know that my page has been submitted for creation.
• When the page I submitted changes state, I receive an email letting me know what happened.
• I receive an email when someone expands an article I created, letting me know what the additions were and who made them.
• When I rate my first article, I receive an email thanking me. (Rating could mean adding a comment)
• When my rating comment gets promoted to the talk page, I get a message letting me know, with a link to the talk page discussion.
• When I rate my 5th article, I receive an email thanking me for my contributions, and encouraging me to try to edit.
• I can opt to receive periodic emails about Wikip/media developments (e.g., new features)
• I can sign up for emails about specific Wikiprojects.
• If I stop editing for x weeks, I receive an email encouraging me to resume editing.
• When a page I recently edited gets a template, I receive an email letting me know what happened in plain language.
• When my talk page gets a template, I receive an email letting me know what happened in plain language.
• When a bot does something, I receive an email letting me know what the bot did, without referring to the bot/requiring me to know what a bot is.

• Email controls:
  • I can opt to receive no emails.
  • I can opt to receive all my emails in a dailly or weekly digest.
  • I can consolidate emails by type (e.g., all emails about users editing my talk page)
  • I can opt out of emails by type (e.g., automatic congrats emails, talk page notifications, etc.)

• We should have ways to ask for emails from users that haven't provided (e.g., user submits a MoodBar comment but hasn't provided an email address).