Notifications/Feature requirements

'''This document is a work in progress. Comments are appreciated but this is not a final draft.'''

This page describes feature requirements for a new notifications system for MediaWiki, code-named Echo. Features below are for the first release of Echo in early 2013. To learn more about Echo, check out this project hub, these testing tips, the user experience page and other related documents.

Overview




Echo is designed to replace and augment existing notification systems on MediaWiki sites, as well as provide significantly more control to both users and developers as to how their notifications are handled, read, and deleted. This new notifications system seeks to unify the delivery of interaction messages in MediaWiki core, through a common API that can provide a uniform interface for users, as well as a scaleable, high-performance platform for developers. For a quick visual overview of this project, check the.

Problems & Solutions
We aim to solve these core problems:
 * There is no central notification system on Media Wiki sites
 * The current ad-hoc approach is inefficient
 * Users are not notified of key events
 * Users are confused by current notices

Echo will be developed to provide these solutions:
 * Provide a unified user experience
 * Help developers add it to their code
 * Promote editor engagement

Guidelines
Here are some general guidelines for providing the best experience for our users.

Every notification should be useful to our users. Quality is key to prevent users from turning off notifications.
 * Focus on quality

Notification should only feature the most relevant events, from the user's perspective -- highlighting just the key items from the watchlist, talkpage and other feeds.
 * Only important events

Notifications should be informative, like a good news story, and report objectively about who did what, when and where.
 * Newsworthy stories

Notifications should be concise, with the basic message under 10 words (up to 20 words if there are multiple user names, long article names or text snippets - which should be truncated after x characters)
 * Keep them short and sweet

Make notifications actionable, but don't overload the user with too many actions. Less is more.
 * Actionable

Don't send more than X notifications per day, if possible, so people don't view this as spam. (under discussion)
 * Daily limits?

Only send notifications to relatively active users. If a user stops visiting the site or clicking on the notifications, we should reduce the flow right away, so it is not perceived as spam. (under discussion)
 * Active users only?

Track how often users click on each type of notifications. If we find that clickthrough falls below a certain threshold (e.g. 15%) for certain types of notifications, we may want to consider removing them from the feed -- or not make them default options in preferences.
 * Measure the impact

These guidelines are still preliminary and subject to revision. Once we refine them some more, they may be useful when communicating with developers to use our Echo API to add notifications for their own extensions.

User groups
For the first release, we will support three types of registered users:
 * new editor (our primary target)
 * active editor
 * very active editor/admin

However, we will focus on new editors (over experienced editors) for this first release, because new users need notifications more than power users. Specifically, we will concentrate on some of the first notifications which a new user will receive after creating an account on Wikipedia, as shown in the workflow to the right.

This doesn't mean we will not support advanced editors, but we will initially emphasize notifications that help engage new users, who need this service most urgently.

Anonymous, unregistered users will not be targeted for this release.

Key features


Key features for Echo's first release include:
 * user menu
 * flyout
 * all-notifications page
 * email notifications
 * preferences page

For an overview of how these features work together, check the (see thumbnail to the right).

In phase 1 of this project (Oct. 2012-Jan. 2013), we will create a simple version of these features, focusing on a first desktop version for both web and email delivery, as described below.

Other features under consideration for later development may be added in separate sections below, including a mobile version, which will not be developed until the second release in spring 2013 (after WMF's mobile platform starts supporting log-in features).

User Menu
The first visible touchpoint for users will be a 'Notifications' badge in the top menu that appears after the user's name in the upper right corner of any web page (referred to below as the 'user menu' or 'growler').

Notifications badge For the first release, we now plan to simply show a 'badge' (also called a 'growler') without a label next to the user name.

Notifications badge If the user has notifications they have not already seen, the badge will be red and will show the number of such 'unseen' notifications. This is likely be a red rectangle with a number ranging from 1 to 999, to match best practices. This badge will auto-update as outlined below, so that if a new notification comes in, the number will be changed as soon as possible. To be clear, the number in the red badge refers to new, unseen notifications that occurred since the last time you looked at the flyout.

Notification flyout Clicking on the "Notifications badge' will display the Notifications Flyout and zero out the badge, turning it gray. Clicking on it again (or anywhere outside of the notifications flyout), will close that flyout, and mark all notifications as read. The badge will then be grayed out, not red and will no longer have a number, until new unseen notifications are added. However, you may still click on that gray badge, which will display the flyout again, with its most recent notifications.

No notifications The gray badge with the number zero would be shown next to the user name, even if there are no new notifications (or no notifications at all), so that the user can still click on it to easily access the flyout, the all-notifications archive or preferences. If the user has not received any notifications at all, the flyout and all-notifications pages would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Flyout


The flyout will feature a short list of new notifications, when the user clicks on the badge in the user menu. See mockup to the right.

Purpose The purpose of the flyout is to:
 * make users aware of new activity that relates to them
 * give them a short list of the most recent notifications
 * enable them to take action on any event in that list
 * view more notifications on the all-notifications page

Number We currently plan to list up to 10 notifications in the flyout, but would only show about 6 notifications at a time, and the user would need to scroll to see the rest. If the user wants to see more, they can click on 'See all' to go to the 'all-notifications' page. It is likely that these numbers will vary between devices, based on the height of your display -- and will be reduced for smaller screens, since scrolling in a flyout is not the best user experience. We will tweak these numbers once we have a first working version of the flyout we can test on various devices.

Order Notifications will be shown in reverse chronological order, with the latest notifications shown first, based on their timestamp. The most recent notifications will remain on the flyout until replaced by newer notifications, which will push them down the stack incrementally, until they fall off from the flyout list. (The same ordering principle will be used in the all-notifications page, except that the list will be longer and separated by days: today, yesterday, all the way back to the past 7 days).

Scrollbar We will provide a scrollbar in the flyout for devices that can support them. This will be needed in order to see up to 10 items in the flyout, which will not all fit at once in the first view of the flyout, particularly if they all have payloads.

Highlights When you first open the flyout, its 'new' notifications will be highlighted, so you can tell at a glance which are new. 'New' means a notification that you haven't seen yet, and that occurred recently, since the last time you opened flyout. To be clear, there should be as many highlighted notifications as the number in the red badge. If you click on a highlighted notification, it becomes 'read' and loses its highlight.

If needed, we may also want to provide a different, grayed-out highlight to indicate items that you have already clicked on previously. Its opacity could be reduced, making it light gray, so you can tell it apart from the ones you have not yet clicked on. However, this is a lower priority than the primary highlight for 'new' items described above.

Types The flyout would include different thypes of notifications, including these sample types now in development:
 * Edit reversions (e.g.: 'Vibha undid your edit to Breakfast')
 * Reviews ('Ryan reviewed a page you started: Breakfast.')
 * Mentions ('Fabrice mentioned you on this page: Dog')

Sample notifications we are working on for the first release are listed below.

Grammar The proposed grammar for each flyout notification follows this general format:

Fabrice Florin posted on your. "Thanks for your great work!" 1 hour ago

This would be based on a common 'subject-verb-object' structure, with a few important attributes, as outlined in this pseudo code;

' : [""] '

Ideally, notifications would help answer these key questions about each event: who? what? where? when? why? how? -- if available in a concise format -- to give the user a better sense of what this event is about. Note that many events will not cover all these answers.

Depending on the type of notification, we may use passive, rather than active voice. For example, we would use the active voice with the subject first for talk page messages or mentions that are related to person-to-person interactions. But for notifications where the object is more important than the actor, we may use passive voice, where the object is named first, then the actor, as shown in this example:

"" was reviewed by Fabrice Florin. Tags: Copy edit 1 hour ago

Examples Here are a few more examples of notifications:

Reviewed and marked for deletion:

[icon] "" was reviewed by Kaldari and marked for deletion. Tags: Unambiguous copyright infringement  1 day ago

Edit undone:

[icon] Your edit to "Breakfast" was by Kaldari.  "This is already covered in the 'Nutrition' section." 1 minute ago'

See more examples in the section below.

Links Each notification will typically contain a single link to the most important action the user can take to respond to the event they are being notified about.

This primary action could link to any of these possible pages: The primary action for each notification type is now described in this prioritized notifications list and will be posted in the section below.
 * your talk page
 * an article page
 * a difference page
 * a discussion page

For information clarity, we will bold the subject, action and object (as shown above), even if they do not have a link attached to them.

Note that the 'all-preferences' page will display more links than the flyout, to provide users with a wider range of actions.

All links from the flyout should open in the same window, not a new window.

Other links The flyout would include a 'See all' link to the 'All Notifications' page.

If space allows, we may also provide one more link to the 'Notifications preferences' page.

We will not display any 'Clear New' button at this time.

Payload When important messages or event details are available in a concise format, we will aim to include short text snippets, to give the user a sense of what they might get if they click on it. But these snippets would be short, and truncated after 140? characters, with three punctuation marks ('...') to indicate there's more.

Timestamp The timestamp will be expressed as a 'time elapsed' indicator (e.g.: 1 min. ago, 2 hours ago, 3 days ago). It may only appear when the user hovers over the notification, to reduce clutter.

No notifications message If the user has not received any notifications at all, the flyout would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Privacy For the record, all personal Echo notifications will be completely private, both in the flyout and all-notifications pages, like all other websites. To clarify, even if a public notification is sent to the user in future releases (e.g. 'a new edition of the Signpost has been published'), the content of that notification will obviously be public (but the fact that you received that notification will remain private).

All notifications


The 'All notifications' page will feature a longer list of notifications, when the user clicks on 'See all' in the flyout. See mockup to the right. (Note: this page is sometimes called 'Archive' in other documents about this project; however, that label is for internal purposes only, and will not be shown to end-users.)

Purpose The purpose of the 'All notifications' page is to give users a full list of all notifications received in the past week, in order to:
 * make them aware of recent events that relate to them
 * find previous events they might have missed or ignored
 * enable them to take a wider range of actions on any event in that list
 * show them how they can change their notification preferences

Sample All-Notifications Page Contents Here is an example of what the All-Notifications page could include (rough draft):

All notifications                                 TODAY [icon] "" was reviewed by . TagS: Copy edits, No references, Stub. 1 min. ago [icon]  posted on "Hey Fabrice, nice work on the Breakfast article!" 1 hour ago JANUARY 28 [icon]  was reviewed and marked for deletion by . Tags: Unambiguous copyright infringement. 1 day ago [icon] Your edit to  was reverted by .  "This is already covered in the 'Nutrition' section." 1 day ago NOVEMBER 27 [icon]  posted on your . "Hi Fabrice, I think you are making progress with the Breakfast article, but I think that you should add a section on Nutrition." ...  2 days ago etc.

Numbers We currently plan to list up notifications for the past 7 days in the 'All notifications' page.

That number will vary from one user to another. For a new user, the total number of notifications may end up being only 10 notifications for that week, but for a more experienced user, it could include over 100 for that same week.

There is no need to include the total number of notifications on this page at this time.

By default, this page would only show about 25 notifications at a time, and the user would need to scroll to the bottom to see the rest (using infinite scrolling or 'More' button). (We will tweak this number once we have a first working version of the the 'All notifications' page that we can test on various devices.)

Order Notifications will be shown in reverse chronological order, with the latest notifications shown first, based on their timestamp. The newest notifications will push back the older ones further down the list.

Sections The 'All notifications' page will be separated into different sections, by days: Today, Yesterday, then actual dates ('Nov. 12'), all the way back to the past 7 days.

Length Notifications will be relatively short in this 'All notifications' page, given that there could be more to list at once. So we will aim to keep them to 1 to 3 lines if possible, so it's easier to scroll down the list and see many of them at a glance.

Highlights It doesn't seem necessary to show highlights to separate 'new' or 'unread' -- unlike on the flyout, where it seems more useful.

Icons The 'All notifications' page would show small icons to identify the different types of notifications (e.g.: edit reversions, page reviews, talk page messages or mentions).

Grammar The proposed grammar for each notification would be consistent with what we are using in the flyout and email copy, though slight variations in language may be needed for each format.

Links While we are planning on offering a single link on the flyout, the 'All notifications' page will provide separate links for the subject and object (and sometimes action), to give more flexibility to the end user on that page (as Facebook and Google do already on their the 'All notifications' pages.)

Here are some examples of how these links might appear on the all-notifications page:

[icon] <Kaldari> posted on your : "Thanks for your great work!" 1 hour ago

[icon] <Breakfast> was reviewed by <Kaldari>. Tags: Copy edit 1 hour ago

[icon] <Breakfast> was reviewed and marked for deletion by <Okeyes>. Tags: Unambiguous copyright infringement. 1 day ago

[icon] Your edit to <Breakfast> was by <Kaldari>. <View Changes> "This is already covered in the 'Nutrition' section." 1 minute ago'

See more examples in the section below.

Here are examples of where these separate links might go to:
 * The link for an actor-name <Ryan> would go to their user page.
 * The link for an article-name <Breakfast> would go to that article.
 * The link for <View Changes> would go to the diff page showing how your revision was changed.
 * The link for <More> would go to the talk page (or other page) showing the full message.

See more examples in the section below.

All links from the all-notifications page should open in the same window, not a new window.

Payload There are two primary types of payload for our first release:
 * text snippets (for talk page messages or reverted edits)
 * tags (for page reviews)

Text snippets: When important messages or event details are available in a concise format, we will aim to display short text snippets in the notifications on the 'All notifications' page, to give the user a sense of what they might get if they click on it. But these snippets will be short, and truncated after 255 characters, with a 'More' link to the page that the snippet is being excerpted from. These text snippets will typically be pulled from the 'edit summaries' (which can be handwritten, automatically generated, or a hybrid of the two), though in some cases it may be possible to pull the first line of the actual message (e.g. when a user adds a new section on a talk page). See more examples in the section below. These snippets should be displayed in between quotes, using a special font style (e.g. gray), to distinguish them from the notification narrative. They should typically be smaller than the main notification message, but not so small that they are hard to read by older users.

Tags: In the case of notifications for page reviews, we will display the specific tags that were added to the page during the review. These tags will be comma-separated and kept short, so that we would only show the first 16? characters of a tag, followed by 3 dots (...) if they exceed that limit. In a later stage, we may provide a link to a flyout that would explain what these notifications mean, as proposed in the second mockup to the right. See specific examples in the section below.

Tag Flyouts In a later stage, we are considering showing a flyout on this page to explain what tags mean, when displayed in notifications, as illustrated in the mockup to the right. We plan to postpone this feature until we have a complete notifications system in place.

Timestamp The timestamp would be expressed as a 'time elapsed' indicator (e.g.: 1 min. ago, 2 hours ago, 3 days ago).

Link to Preferences We will provide on the All-notifications page a link to the Notification preferences, so that people can adjust their settings after seeing the list of their current notifications. This link will simply say 'Preferences'. At a later stage, we may want to consider providing a short summary of which preferences they have currently selected in the upper right corner of the page, so it's easier to understand why they are getting some of these notifications.

All-Notifications URL and Visibility The URL for this All-Notifications page would be very short (e.g.: 'http://en.wikipedia.org/wiki/Special:Notifications'). This page will only visible to logged-in users, and they will only be able to see their own page, not anyone else's. (For debugging purposes, though, we may want to have a temporary way to inspect people's pages, to troubleshoot during development).

No notifications message If the user has not received any notifications at all, the all-notifications pages would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Best Practices These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below:
 * many top sites go back up to a year in their all-notifications page (Facebook only shows 7 days)
 * most sites support up to hundreds of notifications on this page
 * notifications range from only 1 line (Facebook) to up to 9 lines (Google)
 * most sites include text snippets and time stamps for all notifications
 * most sites include either an icon or a photo for all notifications

Email notifications
Email notifications are alerts that are sent to you when important events take place that involve you or your activity on the site.

Purpose The purpose of email notifications is to inform you when something happens that relates to you, even when you are not on the site. These emails will enable you to do the following:
 * learn what just happened (who, what, where, when, why and/or how?).
 * go to the page where the event took place, so you can check it out for yourself.
 * take quick action, if you want to.

Message Types Depending on your preferences, you can receive one of three types of email messages:
 * individual notifications
 * daily digest
 * weekly digest

Note that no messages will be sent if the user checks the 'no email notifications' option in their preferences.

Email Formats Email notifications will eventually be available in two different formats: For the MediaWiki release, we will only support plain text emails, and add HTML format options in the En-Wiki release.
 * plain text (first release on MediaWiki)
 * HTML (next release on En-Wiki)

We will send both plain text and HTML to all users by default, and if their email client doesn't support HTML, it will show plain text instead. There will not be any email format preferences unless we hear a strong user demand for this, to keep preferences to a minimum.

Plain text email notifications
Sample Message - Individual Notification Here is an example of an individual email notification, sent in plain text:

From: Wikipedia <notifications@wikipedia.org> Reply to: Wikipedia <no-reply-notifications@wikipedia.org> To: Fabrice Florin (WMF) <fflorin@wikimedia.org> Subject: You have a new talkpage message Wikipedia user Shiftchange posted on your talk page: "I am trying to expand on hydrocarbons and the environment topic so that users may have more accurate information. Hydrocarbons is a pretty critical educational topic too ..." View more: https://en.wikipedia.org/wiki/Breakfast ________________________________________________ To control which emails we send you, visit: http://en.wikipedia.org/wiki/Special:Preferences#Notifications Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

Message Contents Most email messages will typically include these contents, as illustrated in the sample above:
 * Header
 * From
 * Reply to
 * To
 * Subject


 * Body
 * Event summary
 * Payload (e.g. comment, if any)
 * Call to action
 * Action link


 * Footer
 * Preferences notice (*)
 * Preferences link (*)
 * Company address (*)

(*) The items asterisked above are likely to remain constant in most email messages.

Each of these content types are specified one at a time after the sample message below.

Note that in the case of email digests, multiple notifications may be bundled by category in a single message, which may require dividers in between categories.

Header Contents The email headers will include these fields:

The 'From' field will display the name of the site that is sending the notification, with an email address that includes the word 'notifications', as in this example:
 * From

From: Wikipedia <notifications@wikipedia.org>

The 'Reply to' field will also display the name of the site that is sending the notification, with a different email address for tracking purposes, as in this example:
 * Reply to

Reply to: Wikipedia <no-reply-notifications@wikipedia.org>

The 'To' field will display the userID of the user that is receiving the notification, with the email address they specified in preferences, as in this example:
 * To

To: Fabrice Florin (WMF) <fflorin@wikimedia.org>

The 'Subject' field will feature a short phrase describing the event, as in this example:
 * Subject

You have a new talk page message

The subject will aim to identify the action and the object of this event, while staying short and sweet.

Body Contents The body of the email notification will include these fields:

The 'Event summary' field will consist of a sentence or two describing what happened, who did it, where and when, as in this example:
 * Event summary

Wikipedia user Shiftchange posted on your talk page

or:

Wikipedia user Kaldari reviewed this page: "Breakfast"

Event summaries may vary based on the type of notification. For example, the name of the page will be included when the object is an article, but not if it is a talk page. Some summaries will have more words than others -- and could even include a second phrase, if needed.

Each major group of notifications will share the same phrasing and phrase order, if possible (e.g. all instances of the 'page review' notifications would inherit the same sentence construction).

The 'Payload' field will display relevant information to help understand how or why the event took place, such as tag names for a page review, as in this example:
 * Payload

Tags: Copy edit, No references, Stub

... or, in the case of a talk page post:

Wikipedia user Kaldari posted this on your talk page: "Hey Fabrice, nice work on the Breakfast article! I just linked it to the Nutrition article, and added more info on calories ..."

The payload will not display more than a maximum number of characters (e.g.: 255 characters, as in the all-notifications page). If the payload exceeds that limit, the email notification will only display the first few characters up to that maximum, with three dots at the end, to indicate that it was truncated and that there is more to it. Words should not be truncated, if possible. See the 'All-notifications' section above for more recommendations on payload contents.

See more examples in the section below.

The 'Call to action' and 'action link' fields will will include a short call to action, as well as a separate link shown on a line by itself, as in this example:
 * Call to action and link

View more: https://en.wikipedia.org/wiki/Breakfast

Calls to action and links may vary based on the type of notification. Ideally, the selected link will help users find out what happened and take quick action, if they can and want to.

There should always be a call to action and link, if possible, to invite users to return to the site and participate more actively. We may want to avoid links that exceed 72 characters, if at all possible, so that line wrapping doesn't break the URL on some email clients.

Footer Contents The email footer will include these fields, which will generally stay constant across all notifications.

For the first release, the plain text version of this email footer could be separated from the body of the message by a divider. In future releases, when we implement an HTML version of the email notifications, this email footer could be shown in smaller gray text font, as most other top sites do.

The 'Preferences notice' field will display in small text the email address used for this subscription, along with instructions on how to change their preferences. The 'link' field will enable the user to change their email notifications preferences, and be listed on a line by itself, as in this example:
 * Preferences notice and link

To control the emails we send you, visit: http://en.wikipedia.org/wiki/Special:Preferences#Notifications

This link will take users directly to the preferences for email notifications and will require them to log in.

In future releases, we may consider updating this link to enable the user to immediately unsubscribe with a single click that doesn't require them to log in to change preferences manually.

There should always be at least a preferences link, which should ideally not exceed 72 characters, so that line wrapping doesn't break the URL on some email clients.

This short legal notice with Wikimedia's corporate address will be added at the end of the notifications.
 * Company address

Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

The corporate address is required by law, and implemented by other top sites such as Facebook, Google or LinkedIn.

HTML email notifications
This section is in progress and should be completed shortly.

A preliminary mockup is shown to the right, for discussion purposes.

Bundled Email Digests
Email digests are bundled notifications that are sent daily or weekly.

Digest Goals The goal of email digests is to group notifications into daily or weekly updates, so that:
 * users can get fewer emails sent to them, by getting regular updates instead.
 * users can get an activity summary of what is happening on the site, even if they don't have time to go there.
 * Wikipedia can stay in touch with users and engage them remotely, bringing them back to the site more often.
 * users can select which events to check and take immediate action on the most important ones (HTML only).

Email Frequency The frequency of bundled email digests will be controlled by the user, in their preferences. This will enable them to select either 'daily' or 'weekly'. But the exact time and date will be determined by the Echo software.

Ideally, these digests would be sent at about the same time of day (or the same day of the week), if technically feasible, so that the user can expect to get them somewhat regularly. The actual time will be determined after consultation with engineering and operations teams. If at all possible, it would make sense to send the daily emails in the early mornings for the majority of our target users (e.g.: 3am PT for English Wikipedia). And it would also seem helpful to send the weekly digests on a Monday, at the start of the week. More research will be needed to finalize these proposed times and dates, so we can verify that these proposed assumptions are correct.

Contents The contents of the email digest messages will be roughly the same as the all-notifications archive page. We will not attempt to exclude the notifications that have already been 'read' by the user, at least not for the first release. So it is possible that a user might see some notifications they have seen before, but we think this will be OK, since we are presenting these emails as 'summary of activity'. Categories We propose to group bundled notifications in categories, by notification type, as outlined below.

The rationale for this approach is that it will allow us to show the most important types of notification first, for editor engagement purposes. For example, we would list talk page messages first, to encourage users to interact with others. We would also list positive notifications before negative ones (e.g. edit reverts would be last).

Here are the main categories, listed in order of priority:


 * Talk page message
 * post on your talk page


 * Talk page message (other)
 * reply to your post on another talk page
 * post on the talk page of an article you started


 * WikiLove
 * special message and barnstar on your talk page


 * Mention
 * mention on another talkpage


 * Cross-reference
 * link to a page you started
 * your page was added to a Wikiproject


 * Recognition
 * your page was rated as good
 * your page was featured


 * Page review
 * review of a page you started
 * review with maintenance tags
 * review with deletion tags

(only show this if user opts in for this type of negative notification)
 * Edit revert
 * edit undone
 * edit rolled back

Threshold We will only send up to 10 notifications at a time in a bundled email digest, to prevent the user from being overwhelmed. If the number of notifications exceeds that threshold, the most important notifications would be included in that short list, and the older notifications excluded, if possible. For example, in the case of a daily bundled notification, we would list all the notifications that occurred in the past 24 hours, then sort them by priority (see above order), then show the first 10 notifications from that sorted list, grouped by the categories above.

Plain text email digests
Sample Message - Daily Digest (Bundled Notification) Here is an example of a daily email digest, which is a bundled email notification (preliminary draft):

From: Wikipedia <notifications@wikipedia.org> Reply to: Wikipedia <no-reply-notifications@wikipedia.org> To: Fabrice Florin (WMF) <fflorin@wikimedia.org> Subject: You have 6 notifications today Fabrice, You have 6 notifications on Wikipedia today. View them here: http://en.wikipedia.org/wiki/Special:Notifications 3 Talk page messages • Shiftchange posted on your talk page • Accedie posted on your talk page • Kaldari posted on your talk page ________________________________________________ 2 Page reviews • "Breakfast" was reviewed by Kaldari • "Dehli Metro" was reviewed by Accedie ________________________________________________ 1 Edit revert • Your edit to Nazi Germany was reverted by Kaldari ________________________________________________ To control which emails we send you, visit: http://en.wikipedia.org/wiki/Special:Preferences#Notifications Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

HTML email digests
This section is in progress and should be completed shortly.

Other email specifications
Email Cadence Email frequency can be adjusted to some extent in the user preferences, but is in fact more complex than that, if we want to avoid spamming users. For example, the initial notification about an event should be sent as close to real time as possible, but we may want to batch subsequent notifications if they exceed more than 3? emails in a given day. Sending all notifications immediately could lead the user being overwhelmed and unsubscribing. Sending all notifications batched into x hour bundles may be a reasonable low-effort compromise, but could mean missing an opportunity to drag users back to the site to do more work while things are still fresh. We will need to get the cadence right for a large percentage of users, or risk losing the privilege of communicating with them that way. For the first release, we will implement a simple frequency solution, but will want to tweak its cadence in the second release.

Best Practices These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below: (for practical reasons, we will only implement plain text emails for the first release)
 * most email notifications are HTML, requiring us to support this format as soon as possible
 * the subject line always includes a brief event description and the name of the site
 * average of 3 links per email (min. 2, max. 7 in example slides)
 * 3 out of 5 sites provide a daily and/or weekly digest
 * most sites provide a quick way to unsubscribe

Preferences


The preferences page will feature a short list of options for email and web notifications, as part of the main user preferences.

Purpose The purpose of the preferences is to give users some control over which notifications to receive by email, how often to receive them and whether or not to display the web notifications flyout in the user menu.

Tab A special 'Notifications' tab will be displayed within the main user preferences page. Ideally, it would be listed in second or third position after the first tab you see when you open preferences. We may want to have a one-line explanation of what notifications are below the tab: "We send you notifications when users take actions on Wikipedia that involve you. You can change which notifications to receive below. Learn more >'. ('Learn more' will link to the help page or FAQ where we explain how notifications work.)

Links This notifications preferences tab will be linked from the 'All Notifications' page (and possibly from the 'Notifications' flyout, if space allows).

Sections The first version of preferences would consist of these sections:
 * Email Frequency
 * Email Notifications
 * Web Settings

Email Frequency
This section will enable users to choose how often they want to receive notifications by email, using radio buttons or similar selection method.

Here are the proposed contents for that section:

Email Frequency I would like to receive: • no emails about these events • individual notifications as they come in • a daily summary of most important notifications • a weekly summary of most important notifications Notifications are being sent to myemail@mysite.com. <Change your email address>

Email Notifications
This first section will enable users to choose which notifications to receive by email, using checkboxes.

Here are the proposed contents for that section:

Notification Types Email me when someone: • posts on my talk page • mentions me • send me wiki love • links to a page I started • features a page I started • reviews a page I started • reverts my edit

For now, we are listing the first notification types now in development, but may add more options in the future. From a technical standpoint:
 * 'reverts and edit' includes edits undone and rolled back
 * 'reviews a page' includes all versions of reviewed, including reviewed + tagged, reviewed + marked for deletion
 * 'features a page' includes all versions of rated, featured or today's featured article

Web Settings
This third section will enable users to choose whether or not they want to see the notifications web flyout and badge, using a check box.

Here are the proposed contents for that section:

Web Settings Show badge in the top right menu

By default, this option would be checked. But users who really don't want to see notifications on their browsers would have the option to turn off the flyout and the badge. They would, however, continue to have an 'All-notifications' page, even if it's not visible in their UI.

Web Notifications
After testing the beta version of Echo, many users have requested the option to control which notifications are included in their flyout and archive, in much the same way as they control email notifications. This is something which Wikipedians are likely to be very concerned about, so we may need to provide them with greater control than would be offered on a typical consumer site.

This 'Web Notifications' feature could be offered in much the same way as we do now, by providing multiple check boxes for each option: one for email and one for onsite. Note that this is the method which Google now provides for setting email and phone notifications (see Google notifications settings screenshot).

To that end, we may want to consider these contents for a revised preferences section, which might look a bit like this:


 * Note: For talk page notifications, users could only disable email notifications, not web notifications, because it is important that users be notified if someone leaves a message for them. We may want to display a tooltip over that disabled checkbox to explain this exception ('You cannot disable talk page notifications on the web.'). For similar reasons, users would not have the option to disable general system messages (e.g.: 'your user rights have changed'), because it is critical that users be notified of such important events; therefore, there are no preferences for system messages, for which both web and email notifications would be provided.

Defaults by User Group
To better serve the unique needs of different user groups, we propose separate default settings for new and experienced users. As a rule of thumb, a new user could benefit from more notifications than an experienced user, due to their lower level of activity and need for more guidance.

This could be accomplished by looking up the user's registration date and user group and specifying different settings based on these criteria.

Here are proposed defaults for each user group, for discussion purposes.

New users
 * Defined as users registered in 2013, who do not have special user rights (except for auto-confirmed users, who should be included in this group)
 * Enable individual email notifications
 * Show badge (for onsite notifications)
 * Enable onsite and email notifications for:
 * talkpage posts
 * user mentions
 * wiki love
 * page links
 * page reviews
 * Disable onsite and email notifications for:
 * reverts my edit

Experienced users
 * Defined as users who registered before 2013 and/or have special user rights (e.g.: reviewer, rollbacker, stewart, administrator, etc.)
 * No email notifications (but send one email notification to invite them to opt-in after first month)
 * Show badge for onsite notifications (not shown for first month, while we debug en-wiki)
 * Enable onsite and email notifications for:
 * talkpage posts
 * user mentions
 * wiki love
 * page reviews
 * reverts my edit
 * Disable onsite and email notifications for:
 * page links

The above default settings are proposed here for discussion purposes, and will be revised and expanded based on user feedback.

Other Email Preferences
We may also want to have a section for selecting HTML vs. email later on, as well as reminding people of which email address their notifications are going to, with a link to the preferences that let you set your email address

See All-Notifications It may be helpful to include at the end of this preferences section a link to the all-notifications page: 'See All Notifications.' This would make it easier for users to go back and forth between Preferences and All-notifications to see if they have the right settings.

Best Practices These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below:
 * most settings are about email notifications
 * average of 35 settings, 20 turned on by default
 * typically grouped by object type (e.g.: pages, messages)
 * 3 out of 5 sites we studied provide a daily and/or weekly digest

Dismiss
The 'Dismiss' tool will enable users to turn off notifications they don't want to see in the flyout or the all-notifications page, so they don't have to go to the preferences page if they don't want to.

Purpose The purpose of this feature is to give users more control over the notifications they receive, so they can:
 * quickly turn off notification types they don't want to receive
 * do this in the flyout or the all-notifications page, so they don't have to go to preferences
 * change their preferences so they no longer dismissed notification types, both on the web and by email

Functions Here is how the dismiss feature would work, as shown in the mockups to the right:
 * When the user hovers over a notifications, a gray "(x)" button appears on the right side.
 * When the user clicks on that "x" button, the notification is replaced with "Turn off all 'Edit Reverted' notifications?", with a "Turn off" button and "Cancel" link
 * If the user clicks "Turn off", disable both web and email user preference for that notification type, and show this message: "You have turned off 'Edit Reverted' notifications. <Undo>"
 * If the user clicks "Turn off", show the notification again and do nothing else.
 * If the user clicks "Undo", show the notification again and do nothing else.
 * Once the user turns off a particular notification type, all notifications of this type will be dynamically removed from the flyout and all-notifications page and will no longer appear in the emails either
 * If later on the user wants to re-enable a dismissed notification type, they can do this in preferences

Exceptions The dismiss button would not be available for certain types of notifications which are considered critical and time-sensitive, such as:
 * talk page messages
 * system notifications (e.g. your user rights have changed)
 * onboarding (e.g. welcome, check out your watchlist)

These notifications cannot be dismissed and no 'X' button will appear when you mouse over them in the flyout or all-notifications page.

Best Practices Here are some examples of how this dismiss feature is implemented on other sites like Facebook:
 * Turn off group notifications
 * Turn off app notifications (full sequence)
 * Unfollow comments on your link
 * See also: Facebook Notification API developer guidelines ('User Opt-Out' section)

Bundling
The 'Bundling' feature will enable users to only receive a single notification instead of many notifications about related events, so they are not overloaded with too much information. Bundling aims to address one of the key challenges for this product: how to make sure that we inform people without spamming them.

Purpose The purpose of this feature is to:
 * reduce the number of notifications users receive for related events
 * show a single notification instead, bundling all related notifications into one
 * provide a single link where users can take meaningful action on all bundled notifications

Bundled Notifications Only certain types of notifications can be bundled to meet the above objectives, in order to provide a single link where users can take action on all the bundled notifications. In the current list of notifications under development, these two types seem to be good candidates for bundling:
 * talk page messages
 * page links

Here are examples of how these bundled notifications would be displayed in the flyout:

Tom Morris and 3 others posted on : "Thanks for your good work on the 'Breakfast' article. ..."

(instead of showing several different notifications about posts on your talk page)

or:

'Portland Public Library' was linked by Kaldari and 2 others. <See all links to this page>.

(instead of showing several different notifications about links to your started page -- the bundled notification would point to the special page 'What links here')

A key requirement is that all notifications to be bundled share the same link, so that a user can easily take action on all these notifications from that same page. Developers of new notification types will have access to a special setting where they can enable or disable this bundling method. To prevent spam, this bundling setting will be turned on by default for new notification types, but clear guidelines will be provided to developers about the single link requirement. We may also want to include an automated check that a single link is shared by all bundled notifications, to prevent developers from inadvertently making it hard for users to take action.

We don't expect to bundle other notification types currently in development for the first release, because they are either unlikely to trigger multiple notifications, or can not provide a single link where the user can take meaningful action. That said, we are considering other types of notifications, which might benefit from bundling:
 * '5 editors contributed to <Porcelain money> since your last edit.' (See 'Activity for your recent edit')
 * '<Your watchlist> has 15 new events which may interest you.' (See 'Check out your watchlist')
 * 'Your page on <Higgs Boson> is becoming popular (1k views, 20 comments and linked by 2 other pages today)'
 * 'You have 6 new notifications on <Commons>' (the first cross-wiki notification we expect to implement)

Functional requirements Different functional requirements will be needed for bundling web notifications in the flyout and all-notifications page, as well as for email notifications.

For web notifications in the flyout, functional requirements include:
 * For each notification type that has web bundling enabled, look for all the recent events that occurred in the last 7 days
 * Sort all notification events by link, then group notifications that share the same link, and sort each group by date, only going back to the last 7 days
 * For each group, construct a bundled message that identifies the last user who triggered a notification, then gives the number of other contributors
 * The message will include only one action link for all bundled events on the flyout (e.g. a link to your talk page)
 * If a payload is required for that notification type, only show the most recent text snippet from the last bundled event
 * Show the bundled notification as soon as all messages are ready

For web notifications in the all-notifications page, functional requirements include:
 * For each notification type that has web bundling enabled, follow the same basic requirements in the all-notifications page as for the flyout, but only for the last 7 days
 * For events that occurred earlier than in the past 7 days, we can stop bundling notifications and display individual notifications instead, as outlined below
 * For each event that occurred earlier than in the past 7 days, simply show a notification for each event that identifies the user who triggered it
 * Bundled messages will be constructed in the same format as for the flyout, but links to users and other pages will be added on all-notifications page

For email notifications, functional requirements include:
 * For each notification type that has email bundling enabled, follow the same basic requirements as for web notifications for constructing messages, as outlined above
 * Send the first notification for each notification type and for each link group right away (if individual notifications are set in preferences)
 * If more events occur for that notification type and link group, delay sending a bundled notification for 4 hours (for individual notifications)
 * If even more events occur for that type and group after the first four hour period, send any bundled notifications every four hours (and reset the counter, as needed)
 * For the email digests, list the bundled notifications that took place that day or week for each type and link group, and send them at their regularly scheduled time

For example, here is what a user might receive over the course of a day (with preferences set to individual email notifications):
 * 1pm: 'Tam Valley' was linked by Kaldari.  [single event notification]
 * 5pm: 'Tam Valley' was linked by Tom Morris and 2 others  [bundled notification]
 * 9pm: 'Tam Valley' was linked by BSitu   [single notification]
 * 1am: [no events took place, send nothing, start cycle again]
 * 4am: 'Tam Valley' was linked by Fabrice Florin [single notification]
 * 5am: 'Tam Valley' was linked Okeyes and 1 other [bundled notification]

Note that for maximum flexibility, we propose different settings for web and email bundling (for example, we could enable web bundling for talk page notifications, but disable email bundling to match current user expectations about this important notification type). Also, we propose that these bundled notifications do NOT show links under usernames in the all-notifications page: this greatly reduces the workload for developers, without affecting the user experience significantly.

Dependencies Bundling features require a robust job queue infrastructure, which now does not exist for the current version of Echo.

Best Practices Here are some examples of how this bundling feature is implemented on other sites like Facebook or Google:
 * James, David and 3 others commented on Facebook
 * 4 people added you back on Google+
 * Respond to 20 activities on your page

JobQueue
Various asynchronous backend tasks performed by MediaWiki are added to one or more job queues and popped off those queues and processed. The current JobQueue class stores its data in MySQL tables and uses Memcached for performance and some features. The number of waiting jobs ebbs and flows through the day, but it is not uncommon to have hundreds of thousands waiting and for there to be a significant delay between queueing a job and it getting performed. Jobs will range from relatively cheap tasks like sending a password rest email to fairly expensive ones like transcoding media files.

Adding Echo notifications to the job queue massively increases the number of cheap to process jobs in the queue. It requires, at least for en-wiki, a new type of JobQueue class that can efficiently route notifications to users without load on the database, without waiting in line between longer running, less urgent tasks, and without delaying important tasks that impact many users

The new queue type is likely to be Redis based although this is not a requirement. It should provide a subset of the interface of the current JobQueueDB implementation that is sufficient for Echo's queue needs. Specifically it should provide: * doBatchPush that adds one or more jobs to the queue * doPop that takes a job off for processing.

Management methods like doAck, doIsEmpty and doGetSize are desirable but not required.

Page update specific functionality like doDeduplicateRootJob are not needed as that functionality will stay on the existing queue for the foreseeable future.

Echo Notification Storage
Job queues are not generally persistent, but the system needs to both deliver notifications to users in a timely manner, and also present previously sent notifications for later viewing. This means that as well as putting new notifications into a queue, the system also needs to add them to a persistent mailbox store.

Old notifications are of very limited value, so it is desirable to flush notifications that are more than 30 days old.

Features under consideration
These features are being considered for Echo, though it is not clear yet if they would make it into the first release or future releases. They generally aim to add value for new and experienced editors through a variety of ways (e.g.: by making the watchlist more visible to new users, or by making the notification tools more helpful to experienced users). We aim to investigate these features in coming weeks, and determine which of these can be realistically implemented for the first release, then flesh out their requirements further.

Positive notifications
We would like to provide more positive notifications for the users who have completed an unreverted edit, leading them towards a "happy path" towards further contributions.

Rationale: Oftentimes, new users receive no positive feedback from the community about their first edits, even if they are productive; they only get negative feedback if their edits are reverted. Many studies have shown that positive reinforcement plays an important role in increasing a user's productivity, and some have pointed out that it takes 5 positive notifications to make up for one negative one -- hence the important of such notifications. The assumption here is that under some definition of unreverted, these users are well-intentioned and could become constructive editors.

This problem can be illustrated by this visualization of new user's first edits. Our research suggests that about 74% of a new editor's first actions are article edits, and yet we currently offer no positive notifications to support these first article edits -- only one negative notification ('your edit was reverted').

Here are some of our first ideas for giving positive feedback to new users at the time they need it the most. Please add any other ideas for solving this important issue in the talk page.

Useful edit notification
One possible solution would be to add a 'Mark as useful' button in article history and/or diff pages, to invite experienced editors to mark edits they find useful, so they can send positive feedback to new editors. This could be done through a simple text link next to each edit (e.g. next to 'Undo'): 'Mark as useful' (or 'Useful' for short). When an editor clicks on that link, this notification would be sent to the new editor:

'Kaldari marked your edit as useful on Golden-crowned Sparrow. <View your edit>'

This notification would link to your diff page for that edit. The diff page could include the same message as above, with a link to the experienced editor's talk page, so the new user can start interacting with their supporter, who could eventually become a mentor. If multiple editors mark your edit as useful, we would bundle the notification to say: 'Kaldari and 2 others marked your edit as useful on Golden-crowned Sparrow. <View your edit>', as we do for talk page messages already.

To keep things simple for this experimental feature, we would not keep display the number of 'useful' tags to anyone but the recipient of that notification. Though over time, if the community finds this feature productive, we could expand it with a counter showing the number of positive responses.

This feature would be restricted to logged-in users only, because notifications do not work for anonymous users at this time. So the link would not appear next to anonymous edits. If necessary, we could restrict this feature even more, only to new editors that have fewer than 100 edits (though other more experienced editors might find it valuable as well).

For clarity, we may want to show a small question mark (or flyout) when the user hovers over the 'Mark as useful' link, pointing to a 'Learn more' page explaining how this experimental feature works. We may also provide an 'Unmark as useful' toggle link, in case someone clicks on this by accident.

From a technical standpoint, this feature would be pretty easy to implement in our time frame for the first release, according to our developers. The main question is whether or not our community would agree to this experimental feature. But the outcome for new users is likely to be very constructive, from an editor engagement standpoint.

Huggle/Cluebot Notifications
Another possible solution would be to hook into third-party anti-vandalism tools like Huggle or ClueBot. Some of these tools could be adapted to provide an "approve of edit" button (Huggle already has such a feature), which could send a notification to the user who made the 'good' edit saying something like "this was good! Keep it up".

This would require us to provide a hook (or API) to the third-party developers, but it's actually a lot less work than having to find some other trigger in MediaWiki for it to take its cues off, and has a better signal:noise ratio than any trigger we've yet discussed.

While this seems like a reasonable solution to investigate, it may not be a viable candidate for Echo's first release, because the hooks / API requirements and coordination with third-party developers are likely to take several months to set up.

Contributions since your last edit
One possible solution we have discussed is giving new users a bundled notification that '20 people have contributed to this page since your last edit'. While this type of activity is currently handled by the watchlist, this type of 'contributions' notification could have a positive impact in getting the new user to go back to a page they edited earlier (particularly if they don't yet feel motivated to use the watchlist). This is almost certainly better than only sending them negative notifications when their edits are reverted (which is like a slap in the face) -- or sending them no notifications whatsoever after their first edits (which is what we are doing now). However, this particular notification idea may not always have a positive impact if the user's previous edit was manually reverted or overwritten, which we cannot easily detect, sadly. Also, this would only work for new users with low levels of activity, and could quickly become overwhelming for more active users, as well as strain overall system performance. In order to address this issue, it may be possible to define a threshold that would only send a notification that 'x people have contributed to this page' if x exceeds a certain percentage y of the total number of editors for this page (e.g.: y = 20%? ). Lastly, note that this notification could be based on the number of edits to that page, instead of editors, if that is easier for development purposes.

Active user notifications
We have already identified a number of positive notification ideas for active new users in the [Echo_(Notifications)/Feature_requirements#Positive 'Positive' section below]. They include:
 * WikiLove
 * Started Page -Linked
 * Started Page - WikiProject

While these positive notifications are likely to motivate active new users, a challenge we face is how to give positive reinforcement to new editors immediately after their first few edits, before they become active enough to start new pages or be noticed for a WikiLove message.

Other ideas will be added to this section as we brainstorm them with our team and community.

Watchlist
This section will document interactions between Echo and Watchlist, which are intended to work together as a system for surfacing things that happen on MediaWiki and Wikipedia sites. Our team deliberately identified certain items as out of scope for Echo since they would be covered by watchlist. For example, we didn't see a need to send an Echo notification for every item that appears in the watchlist, to avoid redundancy. In general, we view watchlist as providing ambient awareness about lots of minor events, while Echo is intended to notify the user about important events that impact the user directly.

That said, we would like to make watchlist (and the "notifications" inherent in watchlist) more visible to new users, who may not be aware that this tool even exists, or may not be motivated to check it again, since it is empty when they first register. So we need to at a minimum let new users know that the watchlist exists and potentially make some preference changes to make watchlists easier to use.

Here are some of the possible ways we can ensure that new users are aware that 1) they have a watchlist and/or 2) there are new items in that watchlist. The first two features below seem like likely candidates for the first release, while the others are more likely to be considered for future releases.

How to use your watchlist
This proposed feature would send a one-time notification x days after registration (e.g.: 'How to use your watchlist').

This would offer the following benefits:
 * make the user aware that they have a watchlist
 * provide a link to the watchlist, so they can check it out
 * explain that they can use the star button to add items to their watchlist

The Editor Experimentation team (E3) has expressed interest in developing this notification as part of their onboarding project.

This notification might link to a guided tour on the watchlist page itself -- or a special page explaining how to use the watchlist.

In the flyout, this notification might look like this:

Did you know you have a watchlist? <Learn how it works>.

Here are more parameters about this notification:


 * Trigger: x days after registration
 * Main Link: watchlist page (with a a guided tour?) -- or special page showing
 * Payload: None
 * Frequency: One-time
 * Priority: High (useful to new users)
 * Category: System Message


 * Flyout Summary: [icon] Did you know you have a watchlist? <Learn how it works>.
 * All-notifications Summary: [icon] Did you know you have a watchlist? <Learn how it works>.
 * Email Subject 1: Wikipedia tip: How to use your watchlist
 * Email Summary: Did you know you have a watchlist? <Learn how it works>. [could add a bit more copy if needed]


 * Web preference default: None for system messages
 * Email preference default: None for system messages
 * Web bundling: Disabled
 * Email bundling: Disabled
 * Dismiss feature: Disabled

Special requirements: This notification will only be sent once.

Add new pages to new user watchlists automatically
This feature will automatically add pages that you start or edit to your watchlist, if you are a new user.

The rationale for doing this is to fill in new users' watchlists based on their recent edits, so they can find this tool more valuable and start using it more often.

Here are some open questions we'll want to discuss with the team about this proposed feature:
 * Should we stop adding new pages to a new user's watchlist they become more active? (e.g. after a threshold of, say, 100 edits?)
 * Or should we provide a preference so that the user could stop this from happening on their own?
 * Or should we simply keep filling your watchlist forever?
 * Is it odd that new users from 2013 on get this feature, and not older users)
 * Is this a feature which experienced users might want as well? (a case for having a preference)

Other watchlist ideas
Here are some other ideas which we are considering for future releases.

New events on your watchlist This feature would send a bundled notification informing new users that there is new activity on that page and might look like this:

'You have x new events '

The rationale for this notification is that most new users are not aware that they have a watchlist, or forget that they do, and do not get in the habit of checking it regularly until they become active. This feature would keep nudging them to check it out, and encourage this important habit over time.

This feature is a bit more complex than 'How to use your watchlist' (see above), but would be a good candidate if we have the bandwith to develop more notifications after the initial release.

This could either be a one-time notification as outlined above, or a monthly notification sent for the first three months after notification -- or it could only be sent if they have not visited their watchlist in more than x days, if feasible.

Watchlist Counter A third way is to put a little counter next to the watchlist link in the user menu, so they can see there are new items from any page they visit on Wikipedia.

Watchlist popups Another idea that has been suggested would pop a notification saying there are new events in the watchlist until the new user has clicked n times on the watchlist (default off for experienced editors), until the user builds a habit.

Watchlist page improvements The watchlist page itself could be improved to make it simpler and less overwhelming for new users. Maryana and Vibha have prepared some mockups showing how the watchlist could be made much more usable with just a few simple changes. While this feature would seem like an improvement, it may be hard to get experienced users to let go of a tool which they are already comfortable with. So this would either require that we only make the improvements available to new users (with an opt-in preference for current users), or that we invest some time to negotiate these changes with community members.

Power user notifications
We would like to provide one or more notifications that would be valuable to experienced editors in the first release.

Rationale: the existing feature set for Echo's first release is now targeted mainly at the new user. This is intentional, since experienced users already know where to get information and don't need notifications as much as new users. But it would be nice if we had at least one feature that filled a need for the experienced editor.

Criteria: We plan to select one or two low-risk power-user notifications for the first release, as long as they can match the following criteria:
 * are viewed as really useful by the editor community
 * are feasible, with a low effort in engineering that will not delay the schedule
 * can be used by as many power users as possible over a period of time, so we get more bang for the buck
 * are simple enough that we can manage subsequent feature requests that will inevitably come from experienced editors

In coming weeks, we will to recommend notifications that generally address these criteria and help make this new product useful to as many experienced users as possible.

For example, Oliver has proposed these two notifications, based on IRC conversations with about a dozen power users:
 * user rights notifications ('your user rights have changed - you now have rollbacker privileges')
 * blocked user post ('someone you blocked has left a message on their talk page' -- currently requires administrators to watchlist the talk pages of people they've blocked)

In other conversations, we identified a few more useful notifications which would benefit many active editors:
 * 'your page was rated or featured'
 * 'your page was added to a wikiproject').

You can see some of the notification ideas we discussed at the end of this Google spreadsheet (in the yellow section). The next step is to boil down these various ideas into a short list, and run them by more power users, so we can identify the best prospects for development later this quarter.

Talk Counter
We have discussed the idea of offering a separate counter for the 'Talk' link that appears in the user menu, so that users know the number of unread messages they have on their talk page. The rationale for this proposed feature is to make it easier for people to find out about these personal messages, which could otherwise get lost in a stream of lower-priority notifications. We also discussed adding a flyout for this Talk links, where notifications could be listed separately from the main notifications flyout, but concluded that this would raise too many issues, making it impractical for the first release. For example, people are now used to clicking on the 'Talk' link to go to the talk page, and would resent having to click twice to go there. Also, there will inevitably be some dependencies with 'Flow', our upcoming user-to-user messaging product, and it seems better to hold off on developing a new talk notification features until that new product is further along.

Categories
In future releases, if we determine that users' mental models requires us to separate different types of notification in a single flyout, we may group them by category -- or offer tabs so you could filter the flyout list. If notifications are grouped by category, it is possible that some notifications listed at the top may contain be older than those contained in groups beneath it. For immediate development purposes, we will stay with the current model of stacking all notifications in a single list.

Mark all as read
Provide a function that would let users mark notifications in the flyout as "read". This is particularly useful for power users who get a lot of notifications and want a quick way to get rid of them without paging through their archive, which requires many clicks. We want to investigate how this feature would fit with the current system, and whether the effect would be to remove the highlights or the entire notifications from the flyout (the latter case might be better labeled as a "Clear all" function).

API
Once Echo is further along, we plan to provide an API allowing 3rd parties to send notifications (rather than just extensions using hooks). The rationale is that other developers could expand on some of the basic notification framework we are now working on, without requiring the WMF to develop all new notifications. But this would also require us to develop a set of policies governing their use, to prevent folks from spamming our users. It's going to be a challenge to find the right policies to support both goals, and these policies will require extensive deliberations with the Wikipedia community. This is inevitably going to be a major project, both on the technical and community fronts, and is therefore out of scope for our first release. For now, we will continue to research our options, as well as review best practices from other popular platforms that provide a Notifications API to developers, such as Facebook (see their API overview and design guidelines) and Android. (Note that Facebook tracks whether notifications sent by third parties with their API are meeting certain thresholds in terms of clickthrough and uses that data to weed out low-performing or spammy notifications.)

Sample Notifications
For the first release of Echo, we are currently working on a first set of notifications, which are outlined below and described in more detail on this prioritized notifications list.

As a rule of thumb, we aim to develop notifications that are:
 * interactive: inviting new users to interact with other contributors
 * positive: making new users feel good about participating on Wikipedia
 * frequent: likely to happen regularly, to keep them coming back

Here is a short list of the key types of notifications we are focusing on, divided into different groups (positive, interactive, neutral, negative, infrequent, under investigation).

Interactive
This group includes notifications that invite new users to interact with other users on a regular basis (but that are not consistently positive or negative).

Talkpage Message
This notification occurs when someone posts on your talkpage. This includes events like starting a new section ('new message' or 'template'), or editing an existing section ('respond' or 'cleanup').

<Smallbones> posted on : "Thanks for all your fine work on Wiki Loves Monuments!".


 * Trigger: User edit (click 'Edit' -- or 'New section')
 * Main Link: Talk page (to edited section, if possible)
 * Payload: Show 'Edit summary' (or 'New section first line')
 * Frequency: High
 * Priority: High
 * Category: Talk page message


 * Flyout Summary: [icon] Fabrice Florin posted on
 * Archive Summary: [icon] <VBamba> posted on
 * Email Subject 1: You have a message on your talk page  [generic, from an anonymous user]
 * Email Subject 2: Shiftchange posted on your talk page  [specific, from a registered user]
 * Email Summary: Wikipedia user Shiftchange posted on your talk page.

For interactive notifications such as this one, it seems helpful to include the user name in the email subject line 2 ('Shiftchange posted on your talk page'), instead of a generic subject line 1 ('You have a message on your talk page'). This may help new users recognize other members and form stronger relationships with them as a result. But we would only show a user name for registered users, not for anonymous users (IP addresses): in cases like this, we would use a generic message instead. If this is too complex for the first release, we may start with just the generic message.

For the first release, we're not required to specify different messages for each event type, and can simply use 'post on your talk page' as a general-purpose summary, as proposed above. For the new user we are now targeting, it does not seem necessary to have special messages such as 'started a new section on your talk page', which seem more complicated than newbies need -- and could be inaccurate.

Special requirements For talk page messages supported by Echo, we plan to disable all the yellow talk page notices which now appear when your talk page has been edited. Disabling this current default new messages bar is basically a binary choice, not one dependent on notification type. Also, the yellow bars now tell you whether the talk page edit was one or multiple editors, which is not yet part of the Echo interface. If practical, we would not provide notifications for minor edits, unless they exceed 100 bytes per edit. Other than these caveats, we now expect that the proposed implementation for the first release would cover most of the needs of those who get the current yellow bar.

User Mention
This notification occurs when your username is mentioned by someone else, with a link on any talk page (other than yours):

<Sun Creator> mentioned you on : "I am impressed by the work that <Kaldari> has done on this project."


 * Trigger: User edit including a link to your user page from any talk page other than yours, made by someone other than you
 * Main Link: Talk page where your name was added (go to the edited section that includes your name, if possible)
 * Payload: Show phrase that includes your name, if easy to do (or Edit summary as a backup)
 * Frequency: Medium
 * Priority: High
 * Category: Mention


 * Flyout Summary: [icon] Fabrice Florin mentioned you on
 * Archive Summary: [icon] <VBamba> mentioned you on
 * Email Subject 1: Someone mentioned you on Wikipedia   [generic, from an anonymous user]
 * Email Subject 2: Sun Creator mentioned you on Wikipedia  [specific, from a registered user]
 * Email Summary: Wikipedia user Shiftchange mentioned you on.

Reply to Talkpage Message
This notification occurs when someone replies to a message you left on their talkpage:

<Jimbo> replied to your post on : "You are welcome. I completely agree with you".


 * Trigger: User edit on other talkpage section you started
 * Main Link: Talk page (to edited section, if possible)
 * Payload: Show 'Edit summary'
 * Frequency: High
 * Priority: High
 * Category: Talk page message

Started Page - Talkpage Message
This notification occurs when someone leaves a message on the talk page of a page you started:

<Knowledgekid87> posted on of <Transcarpathian Art Institute>. "Good first stab. Have you considered expanding the history section?"


 * Trigger: User edit on talkpage of article you started
 * Main Link: Talk page (to edited section, if possible)
 * Payload: Show 'Edit summary' (or 'New section first line')
 * Frequency: Medium
 * Priority: Medium
 * Category: Talk page message

Positive
This group includes notifications that can provide positive reinforcement to new users on a regular basis. (Infrequent ones are listed in a separate group).

WikiLove
This notification occurs when someone posts WikiLove on your talkpage:

<Utar> sent you <WikiLove>: "Good call! Thanks for making that revision."


 * Trigger: User WikiLove post
 * Main Link: Talk page (to edited section)
 * Payload: Show first line of message
 * Frequency: Medium
 * Priority: Very High
 * Category: WikiLove


 * Flyout Summary: [icon] Fabrice Florin sent you <WikiLove>
 * Archive Summary: [icon] <VBamba>sent you <WikiLove>
 * Email Subject 1: Someone sent you WikiLove   [generic, from an anonymous user]
 * Email Subject 2: Utar sent you WikiLove  [specific, from a registered user]
 * Email Summary: Wikipedia user Shiftchange mentioned you on.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Started Page - Linked
This notification occurs when a page you started is linked from another page:

<Portland Public Library> was linked by <Peteforsyth>. <See all links to this page>.

or, if multiple links were made to this page since the first one, bundle additional notifications as so:

<Portland Public Library> was linked by <Peteforsyth> and 2 others. <See all links to this page>.


 * Trigger: User edit (linking to page you started)
 * Main Link: 'What links here' page (see example)
 * Payload: None
 * Frequency: Medium
 * Priority: High (positive)
 * Category: Cross-reference


 * Flyout Summary: [icon] Portland Public Library was linked by bsitu. <See all links to this page>.
 * All-notifications Summary: [icon] <Tamalpais Valley> was linked by <Fabrice Florin>. <See all links to this page>.
 * Email Subject 1: A page you started was linked on Wikipedia [generic]
 * Email Subject 2: Life of Pi (film) was linked by Kaldari [specific]
 * Email Summary: <Portland Public Library> was linked by Wikipedia user Vbamba. <See all links to this page>.


 * Web preference default: Enabled for new users, disabled for others
 * Email preference default: Enabled for new users, disabled for others
 * Web bundling: Enabled
 * Email bundling: Enabled
 * Dismiss feature: Enabled (web-only)

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner. For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2. However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. But if this is too complex for the first release, we could start with just the generic subject line 1.

Special requirements: This notification will only be shown/sent for article namespace links, and only for internal links in double brackets. We will not show or send it for talkpage links, transcluded links, redirects or page moves -- or any other page links that could create spam. This notification would be opt-in for existing users, which means that it would only be shown/sent to current users if the check it in their preferences. However, it would be checked by default for new users (see defaults section of the preferences page).

Started Page - WikiProject
This notification occurs when a page you started is added to a Wiki Project:

<Muir Woods> has been added to the <California WikiProject>.


 * Trigger: User edit (to WikiProject page?)
 * Main Link: WikiProject page (to edited section, if possible)
 * Payload: Show 'Edit summary' ?
 * Frequency: Low
 * Priority: High (positive)
 * Category: Cross-reference

Featured Feedback
This notification occurs when feedback you posted is featured by an editor:

Your feedback on <Tamalpais Valley> has been featured by <Kaldari>.


 * Trigger: Your feedback was 'featured' by another user (found useful by an editor, or marked as helpful by a reader)
 * Main Link: Feedback page for this article (showing new post on top of 'featured' list, as we do now for displaying your last post on AFT)
 * Payload: Show your comment from the featured post (or moderator note, if any)
 * Frequency: Medium
 * Priority: High (positive)
 * Category: Feedback


 * Flyout Summary: [icon] Your feedback on <Tamalpais Valley> has been featured by Kaldari.
 * Archive Summary: [icon] Your feedback on <Tamalpais Valley> has been featured by <Kaldari>.
 * Email Subject 1: Your feedback has been featured by Kaldari [generic]
 * Email Subject 2: Your feedback on <Tamalpais Valley> has been featured by Kaldari [specific]
 * Email Summary: Your feedback on <Tamalpais Valley> has been featured by Wikipedia editor Kaldari.

Neutral
This group includes notifications that are not particularly interactive, and not consistently positive or negative.

Started Page - Marked as reviewed
This notification occurs when a page you started is marked as reviewed (and possibly tagged in the process):

<Porcelain money> was reviewed by <Sriharsh1234>.


 * Trigger: Editor review (or patrol) of page you started
 * Main Link: Your page
 * Payload: Tags (if any)
 * Frequency: Low
 * Priority: Medium
 * Category: Page review


 * Flyout Summary: [icon] <Porcelain money> was reviewed by.
 * Archive Summary: [icon] <Tamalpais Valley> was reviewed by <HFung>
 * Email Subject 1: A page you started was reviewed on Wikipedia [generic]
 * Email Subject 2: Life of Pi (film) was reviewed by TheHelpfulOne [specific]
 * Email Summary: <Tamalpais Valley> was reviewed by Wikipedia user Jorm

For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2 ('Life of Pi (film) was reviewed by TheHelpfulOne'). However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. If this is too complex for the first release, we could start with just the generic subject line 1 ('A page you started was reviewed on Wikipedia').

If needed, we could use the same summaries above for all notifications related to 'started page reviews' below. For the first release, it doesn't seem essential to change the wording for each subgroup, if we think the tags tell clearly what the issue is. See sections below for more details.

Note that there are 5 or more possible variations of this type of notification, which are described in more detail on this prioritized notifications list.

Started Page - Maintenance tags
This notification occurs when a page you started is marked as reviewed and also tagged in the process:

<Transcarpathian Art Institute> was reviewed by <QatarStarsLeague>. Tags: Orphan, No categories, Copy edit


 * Trigger: Editor review (or patrol) of page you started -- with tags
 * Main Link: Your page (with templates)
 * Payload: Tags
 * Frequency: Low
 * Priority: Medium
 * Category: Page review


 * Flyout Summary: [icon] <Transcarpathian Art Institute> was tagged by QatarStarsLeague. Tags: ...
 * Archive Summary: [icon] <Transcarpathian Art Institute> was reviewed and tagged by <QatarStarsLeague>. Tags: ...
 * Email Subject 1: A page you started was tagged on Wikipedia [generic]
 * Email Subject 2: Life of Pi (film) was tagged by TheHelpfulOne [specific]
 * Email Summary: <Transcarpathian Art Institute> was reviewed and tagged by QatarStarsLeague. Tags: ...

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Note that there are 72 possible tags that may need to be supported for this notification type.

Started Page - Feedback
This notification occurs when a page you started features new feedback:

<Tamalpais Valley> has new feedback.


 * Trigger: New feedback 'featured' on page you started ('featured' = marked as helpful or useful)
 * Main Link: Feedback page for your page (showing new post on top of 'featured' list)
 * Payload: Show comment from featured post
 * Frequency: Medium
 * Priority: Medium (neutral)
 * Category: Feedback


 * Flyout Summary: [icon] <Tamalpais Valley> has new feedback.
 * Archive Summary: [icon] <Tamalpais Valley> has new feedback by anonymous user <76.181.147.57>.
 * Email Subject 1: A page you started has new feedback [generic]
 * Email Subject 2: <Tamalpais Valley> has new feedback [specific]
 * Email Summary: <Tamalpais Valley> has new feedback by anonymous Wikipedia user 76.181.147.57.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner. For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2. However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. But if this is too complex for the first release, we could start with just the generic subject line 1.

Negative
This group includes notifications that can make new users feel bad about participating on Wikipedia. In general, we recommend that these notifications only be shown if the user 'opts-in' in their preferences.

Started Page - Marked for deletion
This notification occurs when a page you started is marked as reviewed and also nominated for deletion, including one or more deletion tags (AFD / CSD / PROD):

<Government of Kazakhstan Airline> was reviewed and marked for deletion by <Jetstreamer>, who started. "This article may not be suitable for inclusion in Wikipedia, according to Wikipedia's policies and guidelines."


 * Trigger: Editor review (or patrol) of page you started -- with deletion tags
 * Main Link: Your page (with templates)
 * Payload: Tags
 * Frequency: Low
 * Priority: Medium (negative, but important)
 * Category: Page review


 * Flyout Summary: [icon] <Government of Kazakhstan Airline> was marked for deletion by Tchay. Tags: ...
 * Archive Summary: [icon] <Transcarpathian Art Institute> was reviewed and marked for deletion by <Jorm>. Tags: ...
 * Email Subject 1: A page you started was marked for deletion [generic]
 * Email Subject 2: Angry Birds was marked for deletion by TheHelpfulOne [specific]
 * Email Summary: <Angry Birds> was marked for deletion by Wikipedia user QatarStarsLeague. Tags: ...

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Note that there are about a dozen possible tags that may need to be supported for this notification type.

Edit reverted
This notification occurs when your edit was just reverted (undone or rolled back):

Your edit on <Hurricane Sandy> was by <Knowledgekid87>.

Your edit on <Hurricane Sandy> has been reverted by <NewsAndEventsGuy>. "Undid revision 521530151: With what other sentence in the lead is it redundant?"


 * Trigger: Editor reversion (click 'Undo', 'Rollback', manual or automated)
 * Main Link: Diff page (shows difference between your edit and theirs)
 * Payload: Show 'Edit summary'
 * Frequency: High
 * Priority: Low (negative, opt-in)
 * Category: Edit revert


 * Flyout Summary: [icon] Your edit on <Government of Kazakhstan Airline> was reverted by Knowledgekid87.
 * Archive Summary: [icon] <Transcarpathian Art Institute> was reverted by <Eloquence>.
 * Email Subject 1: Your edit was reverted [generic]
 * Email Subject 2: Your edit on Angry Birds was reverted by NewsAndEventsGuy [specific]
 * Email Summary: Your edit on <Angry Birds> was reverted by Wikipedia user NewsAndEventsGuy.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Infrequent Notifications
This group includes notifications that are not likely to happen often for new users -- and therefore could be held back for future releases, even though they can be very positive.

Started Page - Rated
This notification occurs when a page you started receives a Good article rating:

<Angry Birds> was rated as <"Good">.


 * Trigger: Editor adds category to page you started
 * Main Link: Your page (or category page?)
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Recognition

Started Page - Featured
This notification occurs when a page you started is linked from another page:

<Thomas Percy> was just.


 * Trigger: Editor adds category to page you started
 * Main Link: Your page (or category page?)
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Recognition

Started Page - Today's Featured Article
This notification occurs when a page you started is Today's Featured Article:

<Nancy Drew> is <Today's Featured Article>.


 * Trigger: Not sure
 * Main Link: Your page (or Main Page?)
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Recognition

Welcome Page
This notification occurs when a user creates a new account:

Welcome to Wikipedia


 * Trigger: Account creation
 * Main Link: Your profile
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Milestone

Under investigation
This group includes notifications that show potential for engaging users, but that require more research to determine their feasibility.

Answer to your question
This notification occurs when your question is answered (in Help / Teahouse/ Moodbar).

<Kaldari> answered a question you asked in <Teahouse>.


 * Trigger: User edit (in section you started on helpdesk page)
 * Main Link: Helpdesk page (to edited section, if possible)
 * Payload: Show 'Edit summary' (or 'answer first line'?)
 * Frequency: Low

Activity for your recent edit
This notification occurs when other editors contribute to a page you recently edited (and your edit survived).

5 editors contributed to <Porcelain money> since you last edit.


 * Trigger: User edits (with no revert)
 * Main Link: Article page
 * Payload: None
 * Frequency: High

Check out your watchlist
This notification occurs when your watchlist has new activity that relates to you.

<Your watchlist> has 15 new events which may interest you.


 * Trigger: Watchlist events (if you haven't visited watchlist yet)
 * Main Link: Watchlist page
 * Payload: None
 * Frequency: Low (if only shown the first time)

Research Goals
Here are our goals for collecting and analyzing data for the Echo notifications project.

We want to measure how effective Echo is in helping users:
 * learn about activity related to them
 * take action on notifications
 * participate on Wikipedia

We plan to answer these questions through a variety of measurements, some of which would be integrated in the Echo tools. Here are some examples of measurements we could take on, for discussion purposes. After an initial feasibility study and prioritization session, we will identify a few key metrics that we think are required and practical for our first en-wiki deployment.

Target Users
Our initial focus is on new users, but we may want to test the tool with other users as well. Here are the activation options we are considering for testing on the English Wikipedia in early 2013:
 * All users (compare new users vs. active users)
 * New users only (after cut-off date, à la Moodbar)
 * Split test (control group vs. Echo group)

Volume Metrics
Note that some of the breakdowns proposed below (e.g. new vs. active user, or source breakdown) are useful across other types of metrics (e.g. posts vs. views vs. clicks). For example, it would be helpful to have a matrix table showing how many posts, views and clicks were generated by each user group across a range of sources.

Posts
How many notifications are being posted to users in a given week? (but not necessarily viewed)
 * for new users
 * for active users
 * for very active users

Note:
 * Rationale for this request: determine whether specific classes of users receive too many notifications

(This can be done by analyzing echo data, each notification is associated with a user along with triggering timestamp, with user edit count as a way to group new/active/very active users, we will know how many notifications are posted to each user group in a given week. However, a user may increase their edit counts in a very short amount of time, a new user two weeks ago may become an active user today, in this case, we would want to use EvengLogging to track the edit count at the time of triggering the notification)

Views
How many notifications are actually viewed in a given week?
 * on the flyout
 * on the archive page(s)
 * via email (plain text vs. HTML)

Notes:
 * We expect to use EventLogger to track these views
 * We would need to discuss building view tracking into our html email template and a server end to record views
 * Potential privacy concerns for HTML tracking, but OTRS has support for email impressions
 * For users with many pages of archive, how many click past the first page?

(We can track view of flyout by tracking the click on the badge, we can also track the view on archive page on page load, I am not sure if we can track view in emails unless users click on something in the email)

Clicks
How many notifications are being clicked on in a given week?
 * interactive notifications (e.g.: talk page messages, user mentions)
 * positive notifications (e.g.: wiki love, page link, promoted feedback)
 * neutral notifications (e.g.: page reviews, user mentions)
 * negative notifications (e.g.: edit reverts, page deletions)

Notes:
 * Break out clickthrough stats by source (flyout/archive/text-email/html-email)

(All these clicks can be tracked by EventLogging)

Email preferences

 * How many emails are being sent in a week to new/active/very active users?
 * How many users unsubscribed? switched to daily? weekly?
 * For users who disabled or reduced their email settings, how many emails had been sent to them in the past day/week?

Notes:
 * Control for users with authenticated email addresses

Users
How many unique users are clicking on their notifications:
 * every day
 * once a week
 * once a month
 * rarely or never

(We can analyze this by using the data from clicks)

Actions
How many users go on to take these follow-up actions?
 * make an article edit (ns0) -- this is the most important action
 * start a page
 * post a message
 * post feedback
 * upload a file
 * other contributions
 * do nothing

Notes:
 * We need to define what is the scope of a "follow up" action.
 * Since it is hard to determine which actions take place as a direct result of a notification, it might make sense to compare a group of users who are getting Echo notifications with a control group without Echo, then track the overall number of edits, new pages and other contributions from both groups over a period of time.
 * This would let us determine whether or not Echo is helping engage users who receive notifications more than other users.

(It's quite difficult to track user follow-up actions especially when it involves multiple sub-actions. For example, if a user posts on my talk page about an article edit, I get the notification, click on the link in the flyout and get redirected to the talk page, click on the article link in the talk page, then start editing by clicking the edit link, it involves many actions in the middle, any of these actions can be initiated from other places)

Usage
Which notification types are used the most? the least?
 * Welcome
 * Talk page message
 * Started page message
 * Wikilove
 * etc.

Notes:
 * The purpose of this request is to determine whether any of these notifications should be eliminated due to insufficient usage
 * To normalize the data, we may want to focus on the ratio of clickthrough versus frequency (e.g.: measure the number of clicks divided by number of notifications)

Productivity
Which notification types appear to be most productive in terms of follow-up actions?
 * Welcome
 * Talk page message
 * Started page message
 * Wikilove
 * etc.

Usefulness
How useful are notifications to users overall? to new vs. experienced users?
 * Very useful
 * Useful
 * Neutral
 * Not very useful
 * Not useful at all

Notes:
 * This measurement of customer satisfaction could be based on a simple survey shown to users for a short period of time.

General dashboards
We would like some live dashboards showing aggregated data above, dynamically-generated.

Specific dashboards
We would like some specific dashboards for each notification type, so that we (or developers using our API) can track their effectiveness.

User dashboards
We would like some individual user dashboards, so we can observe how typical users use this tool. It would also be useful to look at an average by user type as well (e.g. new user vs. active user average).