Notifications/Feature requirements

'''This document is a work in progress. Comments are appreciated but this is not a final draft.'''

This page describes feature requirements for a new notifications system for Wikipedia, code-named Echo. Features below are for the first release of Echo, with a target date of January 2013. To learn more about Echo, check out this project hub, these testing tips, the user experience page and other related documents.

Overview




Echo is designed to replace and augment existing notification systems on MediaWiki sites, as well as provide significantly more control to both users and developers as to how their notifications are handled, read, and deleted. This new notifications system seeks to unify the delivery of interaction messages in MediaWiki core, through a common API that can provide a uniform interface for users, as well as a scaleable, high-performance platform for developers. For a quick visual overview of this project, check the.

Problems & Solutions
We aim to solve these core problems:
 * There is no central notification system on Media Wiki sites
 * The current ad-hoc approach is inefficient
 * Users are not notified of key events
 * Users are confused by current notices

Echo will be developed to provide these solutions:
 * Provide a unified user experience
 * Help developers add it to their code
 * Promote editor engagement

Guidelines
Here are some general guidelines for providing the best experience for our users.

Every notification should be useful to our users. Quality is key to prevent users from turning off notifications.
 * Focus on quality

Notification should only feature the most relevant events, from the user's perspective -- highlighting just the key items from the watchlist, talkpage and other feeds.
 * Only important events

Notifications should be informative, like a good news story, and report objectively about who did what, when and where.
 * Newsworthy stories

Notifications should be concise, with the basic message under 10 words (up to 20 words if there are multiple user names, long article names or text snippets - which should be truncated after x characters)
 * Keep them short and sweet

Make notifications actionable, but don't overload the user with too many actions. Less is more.
 * Actionable

Don't send more than X notifications per day, if possible, so people don't view this as spam. (under discussion)
 * Daily limits?

Only send notifications to relatively active users. If a user stops visiting the site or clicking on the notifications, we should reduce the flow right away, so it is not perceived as spam. (under discussion)
 * Active users only?

Track how often users click on each type of notifications. If we find that clickthrough falls below a certain threshold (e.g. 15%) for certain types of notifications, we may want to consider removing them from the feed -- or not make them default options in preferences.
 * Measure the impact

These guidelines are still preliminary and subject to revision. Once we refine them some more, they may be useful when communicating with developers to use our Echo API to add notifications for their own extensions.

User groups
For the first release, we will support three types of registered users:
 * new editor (our primary target)
 * active editor
 * very active editor/admin

However, we will focus on new editors (over experienced editors) for this first release, because new users need notifications more than power users. Specifically, we will concentrate on some of the first notifications which a new user will receive after creating an account on Wikipedia, as shown in the workflow to the right.

This doesn't mean we will not support advanced editors, but we will initially emphasize notifications that help engage new users, who need this service most urgently.

Anonymous, unregistered users will not be targeted for this release.

Key features


Key features for Echo's first release include:
 * user menu
 * flyout
 * all-notifications page
 * email notifications
 * preferences page

For an overview of how these features work together, check the (see thumbnail to the right).

In phase 1 of this project (Oct. 2012-Jan. 2013), we will create a simple version of these features, focusing on a first desktop version for both web and email delivery, as described below.

Other features under consideration for later development may be added in separate sections below, including a mobile version, which will not be developed until the second release in spring 2013 (after WMF's mobile platform starts supporting log-in features).

User Menu
The first visible touchpoint for users will be a 'Notifications' badge in the top menu that appears after the user's name in the upper right corner of any web page (referred to below as the 'user menu' or 'growler').

Notifications badge (or 'growler') For the first release, we now plan to simply show a 'badge' (also called a 'growler') without a label next to the user name (even though our first prototype now has a "Notifications" label).

If the user has notifications they have not already seen, the badge will be red and will show the number of such 'unseen' notifications. This is likely be a red rectangle with a number ranging from 1 to 999, to match best practices. This badge will auto-update as outlined below, so that if a new notification comes in, the number will be changed as soon as possible. To be clear, the number in the red badge refers to new, unseen notifications that occurred since the last time you looked at the flyout.

Clicking on the "Notifications badge' will display the Notifications Flyout and zero out the badge, turning it gray. Clicking on it again (or anywhere outside of the notifications flyout), will close that flyout, and mark all notifications as read. The badge will then be grayed out, not red and will no longer have a number, until new unseen notifications are added. However, you may still click on that gray badge, which will display the flyout again, with its most recent notifications.

No notifications The gray badge with the number zero would be shown next to the user name, even if there are no new notifications (or no notifications at all), so that the user can still click on it to easily access the flyout, the all-notifications archive or preferences. If the user has not received any notifications at all, the flyout and all-notifications pages would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Separate counters for Talk and Watchlist Once the basic notifications framework has been developed, we plan to add counters next to the Talk and Watchlist links in the user menu, to indicate how many new items have been added to these feeds. We discussed adding a flyout for these links, but concluded that we wouldn't be able to support these feeds in the same way as we are the main notifications feed, at least not for the first release.

Flyout


The flyout will feature a short list of new notifications, when the user clicks on the badge in the user menu. See mockup to the right.

Purpose The purpose of the flyout is to:
 * make users aware of new activity that relates to them
 * give them a short list of the most recent notifications
 * enable them to take action on any event in that list
 * view more notifications on the all-notifications page

Number We currently plan to list up to 10 notifications in the flyout, but would only show about 6 notifications at a time, and the user would need to scroll to see the rest. If the user wants to see more, they can click on 'See all' to go to the 'all-notifications' page. It is likely that these numbers will vary between devices, based on the height of your display -- and will be reduced for smaller screens, since scrolling in a flyout is not the best user experience. We will tweak these numbers once we have a first working version of the flyout we can test on various devices.

Order Notifications will be shown in reverse chronological order, with the latest notifications shown first, based on their timestamp. The most recent notifications will remain on the flyout until replaced by newer notifications, which will push them down the stack incrementally, until they fall off from the flyout list. (The same ordering principle will be used in the all-notifications page, except that the list will be longer and separated by days: today, yesterday, all the way back to the past 7 days).

Scrollbar We will provide a scrollbar in the flyout for devices that can support them. This will be needed in order to see up to 10 items in the flyout, which will not all fit at once in the first view of the flyout, particularly if they all have payloads.

Highlights When you first open the flyout, its 'new' notifications will be highlighted, so you can tell at a glance which are new. 'New' means a notification that you haven't seen yet, and that occurred recently, since the last time you opened flyout. To be clear, there should be as many highlighted notifications as the number in the red badge. If you click on a highlighted notification, it becomes 'read' and loses its highlight.

If needed, we may also want to provide a different, grayed-out highlight to indicate items that you have already clicked on previously. Its opacity could be reduced, making it light gray, so you can tell it apart from the ones you have not yet clicked on. However, this is a lower priority than the primary highlight for 'new' items described above.

Types The flyout would include different thypes of notifications, including these sample types now in development:
 * Edit reversions (e.g.: 'Vibha undid your edit to Breakfast')
 * Reviews ('Ryan reviewed a page you started: Breakfast.')
 * Mentions ('Fabrice mentioned you on this page: Dog')

Sample notifications we are working on for the first release are listed below.

Grammar The proposed grammar for each flyout notification follows this general format:

Fabrice Florin posted on your. "Thanks for your great work!" 1 hour ago

This would be based on a common 'subject-verb-object' structure, with a few important attributes, as outlined in this pseudo code;

' : [""] '

Ideally, notifications would help answer these key questions about each event: who? what? where? when? why? how? -- if available in a concise format -- to give the user a better sense of what this event is about. Note that many events will not cover all these answers.

Depending on the type of notification, we may use passive, rather than active voice. For example, we would use the active voice with the subject first for talk page messages or mentions that are related to person-to-person interactions. But for notifications where the object is more important than the actor, we may use passive voice, where the object is named first, then the actor, as shown in this example:

"" was reviewed by Fabrice Florin. Tags: Copy edit 1 hour ago

Examples Here are a few more examples of notifications:

Reviewed and marked for deletion:

[icon] "" was reviewed by Kaldari and marked for deletion. Tags: Unambiguous copyright infringement  1 day ago

Edit undone:

[icon] Your edit to "Breakfast" was by Kaldari.  "This is already covered in the 'Nutrition' section." 1 minute ago'

See more examples in the section below.

Links Each notification will typically contain a single link to the most important action the user can take to respond to the event they are being notified about.

This primary action could link to any of these possible pages: The primary action for each notification type is now described in this prioritized notifications list and will be posted in the section below.
 * your talk page
 * an article page
 * a difference page
 * a discussion page

For information clarity, we will bold the subject, action and object (as shown above), even if they do not have a link attached to them.

Note that the 'all-preferences' page will display more links than the flyout, to provide users with a wider range of actions.

All links from the flyout should open in the same window, not a new window.

Other links The flyout would include a 'See all' link to the 'All Notifications' page.

If space allows, we may also provide one more link to the 'Notifications preferences' page.

We will not display any 'Clear New' button at this time.

Payload When important messages or event details are available in a concise format, we will aim to include short text snippets, to give the user a sense of what they might get if they click on it. But these snippets would be short, and truncated after 140? characters, with three punctuation marks ('...') to indicate there's more.

Timestamp The timestamp will be expressed as a 'time elapsed' indicator (e.g.: 1 min. ago, 2 hours ago, 3 days ago). It may only appear when the user hovers over the notification, to reduce clutter.

Bundling We will create a separate set of feature requirements for bundling and de-duping notifications, which may prove to be one of our main design challenges for this project. An example of bundling would be: 'Benny and 5 others edited a page you started.' Stay tuned for more on this topic.

No notifications message If the user has not received any notifications at all, the flyout would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Future Releases In future releases, we may want to add new flyout features, which are tentatively listed below.

Dismiss: Once the first release has been deployed, we may consider adding a 'dismiss' function, which could be done by adding a small 'X' button in the upper right corner of the notification you are currently hovering over, as Facebook does. This could have the effect of displaying a 'Remove' link, which would let you remove that notification from your list. This button could also be extended to provide a 'Turn off' or 'Unsubscribe' link, in cases where you explicitly subscribed to a notification feed. Some of these ideas may be addressed in future releases, based on user feedback.

Talk: Once the first release has been deployed, we may consider separating Talk messages from other notifications, if feasible and appropriate. One idea we considered would be to show a number next to 'Talk', indicating the number of new talk messages, but taking you directly to your Talk page if you click on either 'Talk' or the number. Another option is to provide a separate flyout but were not sure how people would feel about having to click twice to go to their talk page until Flow is ready.

Groups: In future releases, if we determine that users' mental models requires us to separate different types of notification in a single flyout, we may group them by category -- or offer tabs so you could filter the flyout list. If notifications are grouped by category, it is possible that some notifications listed at the top may contain be older than those contained in groups beneath it. For immediate development purposes, we will stay with the current model of stacking all notifications in a single list.

All notifications


The 'All notifications' page will feature a longer list of notifications, when the user clicks on 'See all' in the flyout. See mockup to the right. (Note: this page is sometimes called 'Archive' in other documents about this project; however, that label is for internal purposes only, and will not be shown to end-users.)

Purpose The purpose of the 'All notifications' page is to give users a full list of all notifications received in the past week, in order to:
 * make them aware of recent events that relate to them
 * find previous events they might have missed or ignored
 * enable them to take a wider range of actions on any event in that list
 * show them how they can change their notification preferences

Sample All-Notifications Page Contents Here is an example of what the All-Notifications page could include (rough draft):

All notifications                                 TODAY [icon] "" was reviewed by . TagS: Copy edits, No references, Stub. 1 min. ago [icon]  posted on "Hey Fabrice, nice work on the Breakfast article!" 1 hour ago YESTERDAY [icon]  was reviewed and marked for deletion by . Tags: Unambiguous copyright infringement. 1 day ago [icon] Your edit to  was reverted by .  "This is already covered in the 'Nutrition' section." 1 day ago NOVEMBER 27 [icon]  posted on your . "Hi Fabrice, I think you are making progress with the Breakfast article, but I think that you should add a section on Nutrition." ...  2 days ago etc.

Numbers We currently plan to list up notifications for the past 7 days in the 'All notifications' page.

That number will vary from one user to another. For a new user, the total number of notifications may end up being only 10 notifications for that week, but for a more experienced user, it could include over 100 for that same week.

There is no need to include the total number of notifications on this page at this time.

By default, this page would only show about 25 notifications at a time, and the user would need to scroll to the bottom to see the rest (using infinite scrolling or 'More' button). (We will tweak this number once we have a first working version of the the 'All notifications' page that we can test on various devices.)

Order Notifications will be shown in reverse chronological order, with the latest notifications shown first, based on their timestamp. The newest notifications will push back the older ones further down the list.

Sections The 'All notifications' page will be separated into different sections, by days: Today, Yesterday, then actual dates ('Nov. 12'), all the way back to the past 7 days.

Length Notifications will be relatively short in this 'All notifications' page, given that there could be more to list at once. So we will aim to keep them to 1 to 3 lines if possible, so it's easier to scroll down the list and see many of them at a glance.

Highlights It doesn't seem necessary to show highlights to separate 'new' or 'unread' -- unlike on the flyout, where it seems more useful.

Icons The 'All notifications' page would show small icons to identify the different types of notifications (e.g.: edit reversions, page reviews, talk page messages or mentions).

Grammar The proposed grammar for each notification would be consistent with what we are using in the flyout and email copy, though slight variations in language may be needed for each format.

Links While we are planning on offering a single link on the flyout, the 'All notifications' page will provide separate links for the subject and object (and sometimes action), to give more flexibility to the end user on that page (as Facebook and Google do already on their the 'All notifications' pages.)

Here are some examples of how these links might appear on the all-notifications page:

[icon] <Kaldari> posted on your : "Thanks for your great work!" 1 hour ago

[icon] <Breakfast> was reviewed by <Kaldari>. Tags: Copy edit 1 hour ago

[icon] <Breakfast> was reviewed and marked for deletion by <Okeyes>. Tags: Unambiguous copyright infringement. 1 day ago

[icon] Your edit to <Breakfast> was by <Kaldari>. <View Changes> "This is already covered in the 'Nutrition' section." 1 minute ago'

See more examples in the section below.

Here are examples of where these separate links might go to:
 * The link for an actor-name <Ryan> would go to their user page.
 * The link for an article-name <Breakfast> would go to that article.
 * The link for <View Changes> would go to the diff page showing how your revision was changed.
 * The link for <More> would go to the talk page (or other page) showing the full message.

See more examples in the section below.

All links from the all-notifications page should open in the same window, not a new window.

Payload There are two primary types of payload for our first release:
 * text snippets (for talk page messages or reverted edits)
 * tags (for page reviews)

Text snippets: When important messages or event details are available in a concise format, we will aim to display short text snippets in the notifications on the 'All notifications' page, to give the user a sense of what they might get if they click on it. But these snippets will be short, and truncated after 255 characters, with a 'More' link to the page that the snippet is being excerpted from. These text snippets will typically be pulled from the 'edit summaries' (which can be handwritten, automatically generated, or a hybrid of the two), though in some cases it may be possible to pull the first line of the actual message (e.g. when a user adds a new section on a talk page). See more examples in the section below. These snippets should be displayed in between quotes, using a special font style (e.g. gray), to distinguish them from the notification narrative. They should typically be smaller than the main notification message, but not so small that they are hard to read by older users.

Tags: In the case of notifications for page reviews, we will display the specific tags that were added to the page during the review. These tags will be comma-separated and kept short, so that we would only show the first 16? characters of a tag, followed by 3 dots (...) if they exceed that limit. In a later stage, we may provide a link to a flyout that would explain what these notifications mean, as proposed in the second mockup to the right. See specific examples in the section below.

Tag Flyouts In a later stage, we are considering showing a flyout on this page to explain what tags mean, when displayed in notifications, as illustrated in the mockup to the right. We plan to postpone this feature until we have a complete notifications system in place.

Timestamp The timestamp would be expressed as a 'time elapsed' indicator (e.g.: 1 min. ago, 2 hours ago, 3 days ago).

Link to Preferences We will provide on the All-notifications page a link to the Notification preferences, so that people can adjust their settings after seeing the list of their current notifications. This link will simply say 'Preferences'. At a later stage, we may want to consider providing a short summary of which preferences they have currently selected in the upper right corner of the page, so it's easier to understand why they are getting some of these notifications.

All-Notifications URL and Visibility The URL for this All-Notifications page would be very short (e.g.: 'http://en.wikipedia.org/wiki/Special:Notifications'). This page will only visible to logged-in users, and they will only be able to see their own page, not anyone else's. (For debugging purposes, though, we may want to have a temporary way to inspect people's pages, to troubleshoot during development).

No notifications message If the user has not received any notifications at all, the all-notifications pages would display this line of text where the notifications would have appeared: "You have no notifications at this time."

Bundling We will create a separate set of feature requirements for bundling and de-duping notifications, which may prove to be one of our main design challenges for this project. An example of bundling would be: 'Benny and 5 others edited a page you started.' Stay tuned for more on this topic.

Best Practices These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below:
 * many top sites go back up to a year in their all-notifications page (Facebook only shows 7 days)
 * most sites support up to hundreds of notifications on this page
 * notifications range from only 1 line (Facebook) to up to 9 lines (Google)
 * most sites include text snippets and time stamps for all notifications
 * most sites include either an icon or a photo for all notifications

Email notifications
Email notifications are alerts that are sent to you when important events take place that involve you or your activity on the site.

Purpose The purpose of email notifications is to inform you when something happens that relates to you, even when you are not on the site. These emails will enable you to do the following:
 * learn what just happened (who, what, where, when, why and/or how?).
 * go to the page where the event took place, so you can check it out for yourself.
 * take quick action, if you want to.

Message Types Depending on your preferences, you can receive one of three types of email messages:
 * individual notifications
 * daily digest
 * weekly digest

Note that no messages will be sent if the user checks the 'no email notifications' option in their preferences.

Email Formats Email notifications will eventually be available in two different formats: For the MediaWiki release, we will only support plain text emails, and add HTML format options in the En-Wiki release.
 * plain text (first release on MediaWiki)
 * HTML (next release on En-Wiki)

We will send both plain text and HTML to all users by default, and if their email client doesn't support HTML, it will show plain text instead. There will not be any email format preferences unless we hear a strong user demand for this, to keep preferences to a minimum.

Plain text email notifications
Sample Message - Individual Notification Here is an example of an individual email notification, sent in plain text:

From: Wikipedia <notifications@wikipedia.org> Reply to: Wikipedia <no-reply-notifications@wikipedia.org> To: Fabrice Florin (WMF) <fflorin@wikimedia.org> Subject: You have a new talkpage message Wikipedia user Shiftchange posted on your talk page: "I am trying to expand on hydrocarbons and the environment topic so that users may have more accurate information. Hydrocarbons is a pretty critical educational topic too ..." View more: https://en.wikipedia.org/wiki/Breakfast ________________________________________________ To control which emails we send you, visit: http://en.wikipedia.org/wiki/Special:Preferences#Notifications Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

Message Contents Most email messages will typically include these contents, as illustrated in the sample above:
 * Header
 * From
 * Reply to
 * To
 * Subject


 * Body
 * Event summary
 * Payload (e.g. comment, if any)
 * Call to action
 * Action link


 * Footer
 * Preferences notice (*)
 * Preferences link (*)
 * Company address (*)

(*) The items asterisked above are likely to remain constant in most email messages.

Each of these content types are specified one at a time after the sample message below.

Note that in the case of email digests, multiple notifications may be bundled by category in a single message, which may require dividers in between categories.

Header Contents The email headers will include these fields:

The 'From' field will display the name of the site that is sending the notification, with an email address that includes the word 'notifications', as in this example:
 * From

From: Wikipedia <notifications@wikipedia.org>

The 'Reply to' field will also display the name of the site that is sending the notification, with a different email address for tracking purposes, as in this example:
 * Reply to

Reply to: Wikipedia <no-reply-notifications@wikipedia.org>

The 'To' field will display the userID of the user that is receiving the notification, with the email address they specified in preferences, as in this example:
 * To

To: Fabrice Florin (WMF) <fflorin@wikimedia.org>

The 'Subject' field will feature a short phrase describing the event, as in this example:
 * Subject

You have a new talk page message

The subject will aim to identify the action and the object of this event, while staying short and sweet.

Body Contents The body of the email notification will include these fields:

The 'Event summary' field will consist of a sentence or two describing what happened, who did it, where and when, as in this example:
 * Event summary

Wikipedia user Shiftchange posted on your talk page

or:

Wikipedia user Kaldari reviewed this page: "Breakfast"

Event summaries may vary based on the type of notification. For example, the name of the page will be included when the object is an article, but not if it is a talk page. Some summaries will have more words than others -- and could even include a second phrase, if needed.

Each major group of notifications will share the same phrasing and phrase order, if possible (e.g. all instances of the 'page review' notifications would inherit the same sentence construction).

The 'Payload' field will display relevant information to help understand how or why the event took place, such as tag names for a page review, as in this example:
 * Payload

Tags: Copy edit, No references, Stub

... or, in the case of a talk page post:

Wikipedia user Kaldari posted this on your talk page: "Hey Fabrice, nice work on the Breakfast article! I just linked it to the Nutrition article, and added more info on calories ..."

The payload will not display more than a maximum number of characters (e.g.: 255 characters, as in the all-notifications page). If the payload exceeds that limit, the email notification will only display the first few characters up to that maximum, with three dots at the end, to indicate that it was truncated and that there is more to it. Words should not be truncated, if possible. See the 'All-notifications' section above for more recommendations on payload contents.

See more examples in the section below.

The 'Call to action' and 'action link' fields will will include a short call to action, as well as a separate link shown on a line by itself, as in this example:
 * Call to action and link

View more: https://en.wikipedia.org/wiki/Breakfast

Calls to action and links may vary based on the type of notification. Ideally, the selected link will help users find out what happened and take quick action, if they can and want to.

There should always be a call to action and link, if possible, to invite users to return to the site and participate more actively. We may want to avoid links that exceed 72 characters, if at all possible, so that line wrapping doesn't break the URL on some email clients.

Footer Contents The email footer will include these fields, which will generally stay constant across all notifications.

For the first release, the plain text version of this email footer could be separated from the body of the message by a divider. In future releases, when we implement an HTML version of the email notifications, this email footer could be shown in smaller gray text font, as most other top sites do.

The 'Preferences notice' field will display in small text the email address used for this subscription, along with instructions on how to change their preferences. The 'link' field will enable the user to change their email notifications preferences, and be listed on a line by itself, as in this example:
 * Preferences notice and link

To control the emails we send you, visit: http://en.wikipedia.org/wiki/Special:Preferences#Notifications

This link will take users directly to the preferences for email notifications and will require them to log in.

In future releases, we may consider updating this link to enable the user to immediately unsubscribe with a single click that doesn't require them to log in to change preferences manually.

There should always be at least a preferences link, which should ideally not exceed 72 characters, so that line wrapping doesn't break the URL on some email clients.

This short legal notice with Wikimedia's corporate address will be added at the end of the notifications.
 * Company address

Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

The corporate address is required by law, and implemented by other top sites such as Facebook, Google or LinkedIn.

HTML email notifications
This section is in progress and should be completed shortly.

A preliminary mockup is shown to the right, for discussion purposes.

Bundled Email Digests
Email digests are bundled notifications that are sent daily or weekly.

Digest Goals The goal of email digests is to group notifications into daily or weekly updates, so that:
 * users can get fewer emails sent to them, by getting regular updates instead.
 * users can get an activity summary of what is happening on the site, even if they don't have time to go there.
 * Wikipedia can stay in touch with users and engage them remotely, bringing them back to the site more often.
 * users can select which events to check and take immediate action on the most important ones (HTML only).

Email Frequency The frequency of bundled email digests will be controlled by the user, in their preferences. This will enable them to select either 'daily' or 'weekly'. But the exact time and date will be determined by the Echo software.

Ideally, these digests would be sent at about the same time of day (or the same day of the week), if technically feasible, so that the user can expect to get them somewhat regularly. The actual time will be determined after consultation with engineering and operations teams. If at all possible, it would make sense to send the daily emails in the early mornings for the majority of our target users (e.g.: 3am PT for English Wikipedia). And it would also seem helpful to send the weekly digests on a Monday, at the start of the week. More research will be needed to finalize these proposed times and dates, so we can verify that these proposed assumptions are correct.

Contents The contents of the email digest messages will be roughly the same as the all-notifications archive page. We will not attempt to exclude the notifications that have already been 'read' by the user, at least not for the first release. So it is possible that a user might see some notifications they have seen before, but we think this will be OK, since we are presenting these emails as 'summary of activity'. Categories We propose to group bundled notifications in categories, by notification type, as outlined below.

The rationale for this approach is that it will allow us to show the most important types of notification first, for editor engagement purposes. For example, we would list talk page messages first, to encourage users to interact with others. We would also list positive notifications before negative ones (e.g. edit reverts would be last).

Here are the main categories, listed in order of priority:


 * Talk page message
 * post on your talk page


 * Talk page message (other)
 * reply to your post on another talk page
 * post on the talk page of an article you started


 * WikiLove
 * special message and barnstar on your talk page


 * Mention
 * mention on another talkpage


 * Cross-reference
 * link to a page you started
 * your page was added to a Wikiproject


 * Recognition
 * your page was rated as good
 * your page was featured


 * Page review
 * review of a page you started
 * review with maintenance tags
 * review with deletion tags

(only show this if user opts in for this type of negative notification)
 * Edit revert
 * edit undone
 * edit rolled back

Threshold We will only send up to 10 notifications at a time in a bundled email digest, to prevent the user from being overwhelmed. If the number of notifications exceeds that threshold, the most important notifications would be included in that short list, and the older notifications excluded, if possible. For example, in the case of a daily bundled notification, we would list all the notifications that occurred in the past 24 hours, then sort them by priority (see above order), then show the first 10 notifications from that sorted list, grouped by the categories above.

Plain text email digests
Sample Message - Daily Digest (Bundled Notification) Here is an example of a daily email digest, which is a bundled email notification (preliminary draft):

From: Wikipedia <notifications@wikipedia.org> Reply to: Wikipedia <no-reply-notifications@wikipedia.org> To: Fabrice Florin (WMF) <fflorin@wikimedia.org> Subject: You have 6 notifications today Fabrice, You have 6 notifications on Wikipedia today. View them here: http://en.wikipedia.org/wiki/Special:Notifications 3 Talk page messages • Shiftchange posted on your talk page • Accedie posted on your talk page • Kaldari posted on your talk page ________________________________________________ 2 Page reviews • "Breakfast" was reviewed by Kaldari • "Dehli Metro" was reviewed by Accedie ________________________________________________ 1 Edit revert • Your edit to Nazi Germany was reverted by Kaldari ________________________________________________ To control which emails we send you, visit: http://en.wikipedia.org/wiki/Special:Preferences#Notifications Wikimedia Foundation, 149 New Montgomery St., 3rd Fl., San Francisco, CA 94105.

HTML email digests
This section is in progress and should be completed shortly.

Other email specifications
Email Cadence Email frequency can be adjusted to some extent in the user preferences, but is in fact more complex than that, if we want to avoid spamming users. For example, the initial notification about an event should be sent as close to real time as possible, but we may want to batch subsequent notifications if they exceed more than 3? emails in a given day. Sending all notifications immediately could lead the user being overwhelmed and unsubscribing. Sending all notifications batched into x hour bundles may be a reasonable low-effort compromise, but could mean missing an opportunity to drag users back to the site to do more work while things are still fresh. We will need to get the cadence right for a large percentage of users, or risk losing the privilege of communicating with them that way. For the first release, we will implement a simple frequency solution, but will want to tweak its cadence in the second release.

Best Practices These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below: (for practical reasons, we will only implement plain text emails for the first release)
 * most email notifications are HTML, requiring us to support this format as soon as possible
 * the subject line always includes a brief event description and the name of the site
 * average of 3 links per email (min. 2, max. 7 in example slides)
 * 3 out of 5 sites provide a daily and/or weekly digest
 * most sites provide a quick way to unsubscribe

Preferences


The preferences page will feature a short list of options for email and web notifications, as part of the main user preferences.

Purpose The purpose of the preferences is to give users some control over which notifications to receive by email, how often to receive them and whether or not to display the web notifications flyout in the user menu.

Tab A special 'Notifications' tab will be displayed within the main user preferences page. Ideally, it would be listed in second or third position after the first tab you see when you open preferences. We may want to have a one-line explanation of what notifications are below the tab: "We send you notifications when users take actions on Wikipedia that involve you. You can change which notifications to receive below. Learn more >'. ('Learn more' will link to the help page or FAQ where we explain how notifications work.)

Links This notifications preferences tab will be linked from the 'All Notifications' page (and possibly from the 'Notifications' flyout, if space allows).

Sections The first version of preferences would consist of these sections:
 * Email Frequency
 * Email Notifications
 * Web Settings

Email Frequency This section will enable users to choose how often they want to receive notifications by email, using radio buttons or similar selection method.

Here are the proposed contents for that section:

Email Frequency I would like to receive: • no emails about these events • individual notifications as they come in • a daily summary of most important notifications • a weekly summary of most important notifications Notifications are being sent to myemail@mysite.com. <Change your email address>

Email Notifications This first section will enable users to choose which notifications to receive by email, using checkboxes.

Here are the proposed contents for that section:

Notification Types Email me when someone: • posts on my talk page • mentions me • send me wiki love • links to a page I started • features a page I started • reviews a page I started • reverts my edit

For now, we are listing the first notification types now in development, but may add more options in the future. From a technical standpoint:
 * 'reverts and edit' includes edits undone and rolled back
 * 'reviews a page' includes all versions of reviewed, including reviewed + tagged, reviewed + marked for deletion
 * 'features a page' includes all versions of rated, featured or today's featured article

Web Settings This third section will enable users to choose whether or not they want to see the notifications web flyout and badge, using a check box.

Here are the proposed contents for that section:

Web Settings Show badge in the top right menu

By default, this option would be checked. But users who really don't want to see notifications on their browsers would have the option to turn off the flyout and the badge. They would, however, continue to have an 'All-notifications' page, even if it's not visible in their UI.

Other Email Preferences We may also want to have a section for selecting HTML vs. email later on, as well as reminding people of which email address their notifications are going to, with a link to the preferences that let you set your email address

See All-Notifications It may be helpful to include at the end of this preferences section a link to the all-notifications page: 'See All Notifications.' This would make it easier for users to go back and forth between Preferences and All-notifications to see if they have the right settings.

Best Practices These feature requirements are inspired in part by best practices used by other top sites or platforms (e.g.: Facebook, Twitter, Google, LinkedIn, Quora), as shown in these slides, and summarized below:
 * most settings are about email notifications
 * average of 35 settings, 20 turned on by default
 * typically grouped by object type (e.g.: pages, messages)
 * 3 out of 5 sites we studied provide a daily and/or weekly digest

Sample Notifications
For the first release of Echo, we are currently working on a first set of notifications, which are outlined below and described in more detail on this prioritized notifications list.

As a rule of thumb, we aim to develop notifications that are:
 * interactive: inviting new users to interact with other contributors
 * positive: making new users feel good about participating on Wikipedia
 * frequent: likely to happen regularly, to keep them coming back

Here is a short list of the key types of notifications we are focusing on, divided into different groups (positive, interactive, neutral, negative, infrequent, under investigation).

Interactive
This group includes notifications that invite new users to interact with other users on a regular basis (but that are not consistently positive or negative).

Talkpage Message
This notification occurs when someone posts on your talkpage. This includes events like starting a new section ('new message' or 'template'), or editing an existing section ('respond' or 'cleanup').

<Smallbones> posted on : "Thanks for all your fine work on Wiki Loves Monuments!".


 * Trigger: User edit (click 'Edit' -- or 'New section')
 * Main Link: Talk page (to edited section, if possible)
 * Payload: Show 'Edit summary' (or 'New section first line')
 * Frequency: High
 * Priority: High
 * Category: Talk page message


 * Flyout Summary: [icon] Fabrice Florin posted on
 * Archive Summary: [icon] <VBamba> posted on
 * Email Subject 1: You have a message on your talk page  [generic, from an anonymous user]
 * Email Subject 2: Shiftchange posted on your talk page  [specific, from a registered user]
 * Email Summary: Wikipedia user Shiftchange posted on your talk page.

For interactive notifications such as this one, it seems helpful to include the user name in the email subject line 2 ('Shiftchange posted on your talk page'), instead of a generic subject line 1 ('You have a message on your talk page'). This may help new users recognize other members and form stronger relationships with them as a result. But we would only show a user name for registered users, not for anonymous users (IP addresses): in cases like this, we would use a generic message instead. If this is too complex for the first release, we may start with just the generic message.

For the first release, we're not required to specify different messages for each event type, and can simply use 'post on your talk page' as a general-purpose summary, as proposed above. For the new user we are now targeting, it does not seem necessary to have special messages such as 'started a new section on your talk page', which seem more complicated than newbies need -- and could be inaccurate.

Special requirements For talk page messages supported by Echo, we plan to disable all the yellow talk page notices which now appear when your talk page has been edited. Disabling this current default new messages bar is basically a binary choice, not one dependent on notification type. Also, the yellow bars now tell you whether the talk page edit was one or multiple editors, which is not yet part of the Echo interface. If practical, we would not provide notifications for minor edits, unless they exceed 100 bytes per edit. Other than these caveats, we now expect that the proposed implementation for the first release would cover most of the needs of those who get the current yellow bar.

User Mention
This notification occurs when your username is mentioned with a link on a talk page (other than yours):

<Sun Creator> mentioned you on : "I am impressed by the work that Kaldari has done on this project."


 * Trigger: User edit including your name and a link to your user page on other talk page
 * Main Link: Talk page where your name was added (to edited section, if possible)
 * Payload: Show phrase that includes your name? (or Edit summary)
 * Frequency: High
 * Priority: High
 * Category: Mention


 * Flyout Summary: [icon] Fabrice Florin mentioned you on
 * Archive Summary: [icon] <VBamba> mentioned you on
 * Email Subject 1: Someone mentioned you on Wikipedia   [generic, from an anonymous user]
 * Email Subject 2: Sun Creator mentioned you on Wikipedia  [specific, from a registered user]
 * Email Summary: Wikipedia user Shiftchange mentioned you on.

Reply to Talkpage Message
This notification occurs when someone replies to a message you left on their talkpage:

<Jimbo> replied to your post on : "You are welcome. I completely agree with you".


 * Trigger: User edit on other talkpage section you started
 * Main Link: Talk page (to edited section, if possible)
 * Payload: Show 'Edit summary'
 * Frequency: High
 * Priority: High
 * Category: Talk page message

Started Page - Talkpage Message
This notification occurs when someone leaves a message on the talk page of a page you started:

<Knowledgekid87> posted on of <Transcarpathian Art Institute>. "Good first stab. Have you considered expanding the history section?"


 * Trigger: User edit on talkpage of article you started
 * Main Link: Talk page (to edited section, if possible)
 * Payload: Show 'Edit summary' (or 'New section first line')
 * Frequency: Medium
 * Priority: Medium
 * Category: Talk page message

Positive
This group includes notifications that can provide positive reinforcement to new users on a regular basis. (Infrequent ones are listed in a separate group).

WikiLove
This notification occurs when someone posts WikiLove on your talkpage:

<Utar> sent you <WikiLove>: "Good call! Thanks for making that revision."


 * Trigger: User WikiLove post
 * Main Link: Talk page (to edited section)
 * Payload: Show first line of message
 * Frequency: Medium
 * Priority: Very High
 * Category: WikiLove


 * Flyout Summary: [icon] Fabrice Florin sent you <WikiLove>
 * Archive Summary: [icon] <VBamba>sent you <WikiLove>
 * Email Subject 1: Someone sent you WikiLove   [generic, from an anonymous user]
 * Email Subject 2: Utar sent you WikiLove  [specific, from a registered user]
 * Email Summary: Wikipedia user Shiftchange mentioned you on.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Started Page - Linked
This notification occurs when a page you started is linked from another page:

<Portland Public Library> was linked by <Peteforsyth> from this page: <Oregon Landmarks>.


 * Trigger: User edit (linking to page you started)
 * Main Link: Page that links to your page (to edited section, if possible)
 * Payload: Show 'Edit summary' (or 'New section first line')
 * Frequency: Medium
 * Priority: High (positive)
 * Category: Cross-reference


 * Flyout Summary: [icon] Portland Public Library was linked by bsitu from this page: <Oregon Landmarks>.
 * Archive Summary: [icon] <Tamalpais Valley> was linked by <Fabrice Florin> from this page: <Transcarpathian Art Institute>
 * Email Subject 1: A page you started was linked on Wikipedia [generic]
 * Email Subject 2: Life of Pi (film) was linked by Kaldari [specific]
 * Email Summary: <Portland Public Library> was linked by Wikipedia user Vbamba, from this page: <Porcelain money>.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner. For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2. However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. But if this is too complex for the first release, we could start with just the generic subject line 1.

Started Page - WikiProject
This notification occurs when a page you started is added to a Wiki Project:

<Muir Woods> has been added to the <California WikiProject>.


 * Trigger: User edit (to WikiProject page?)
 * Main Link: WikiProject page (to edited section, if possible)
 * Payload: Show 'Edit summary' ?
 * Frequency: Low
 * Priority: High (positive)
 * Category: Cross-reference

Your Feedback - Featured
This notification occurs when feedback you posted is featured:

Your feedback on <Tamalpais Valley> is now featured.


 * Trigger: Your feedback was 'featured' by another user (found useful by an editor, or marked as helpful by a reader)
 * Main Link: Feedback page for your page (showing new post on top of 'featured' list)
 * Payload: Show your comment from featured post (or moderator note, if any)
 * Frequency: Medium
 * Priority: High (positive)
 * Category: Feedback


 * Flyout Summary: [icon] Your feedback on <Tamalpais Valley> is now featured.
 * Archive Summary: [icon] Your feedback on <Tamalpais Valley> has been featured by <Kaldari>.
 * Email Subject 1: Your feedback has been featured [generic]
 * Email Subject 2: Your feedback on <Tamalpais Valley> has been featured [specific]
 * Email Summary: Your feedback on <Tamalpais Valley> has been featured by Wikipedia editor Kaldari.

Neutral
This group includes notifications that are not particularly interactive, and not consistently positive or negative.

Started Page - Marked as reviewed
This notification occurs when a page you started is marked as reviewed (and possibly tagged in the process):

<Porcelain money> was reviewed by <Sriharsh1234>.


 * Trigger: Editor review (or patrol) of page you started
 * Main Link: Your page
 * Payload: Tags (if any)
 * Frequency: Low
 * Priority: Medium
 * Category: Page review


 * Flyout Summary: [icon] <Porcelain money> was reviewed by.
 * Archive Summary: [icon] <Tamalpais Valley> was reviewed by <HFung>
 * Email Subject 1: A page you started was reviewed on Wikipedia [generic]
 * Email Subject 2: Life of Pi (film) was reviewed by TheHelpfulOne [specific]
 * Email Summary: <Tamalpais Valley> was reviewed by Wikipedia user Jorm

For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2 ('Life of Pi (film) was reviewed by TheHelpfulOne'). However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. If this is too complex for the first release, we could start with just the generic subject line 1 ('A page you started was reviewed on Wikipedia').

If needed, we could use the same summaries above for all notifications related to 'started page reviews' below. For the first release, it doesn't seem essential to change the wording for each subgroup, if we think the tags tell clearly what the issue is. See sections below for more details.

Note that there are 5 or more possible variations of this type of notification, which are described in more detail on this prioritized notifications list.

Started Page - Maintenance tags
This notification occurs when a page you started is marked as reviewed and also tagged in the process:

<Transcarpathian Art Institute> was reviewed by <QatarStarsLeague>. Tags: Orphan, No categories, Copy edit


 * Trigger: Editor review (or patrol) of page you started -- with tags
 * Main Link: Your page (with templates)
 * Payload: Tags
 * Frequency: Low
 * Priority: Medium
 * Category: Page review


 * Flyout Summary: [icon] <Transcarpathian Art Institute> was tagged by QatarStarsLeague. Tags: ...
 * Archive Summary: [icon] <Transcarpathian Art Institute> was reviewed and tagged by <QatarStarsLeague>. Tags: ...
 * Email Subject 1: A page you started was tagged on Wikipedia [generic]
 * Email Subject 2: Life of Pi (film) was tagged by TheHelpfulOne [specific]
 * Email Summary: <Transcarpathian Art Institute> was reviewed and tagged by QatarStarsLeague. Tags: ...

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Note that there are 72 possible tags that may need to be supported for this notification type.

Started Page - Feedback
This notification occurs when a page you started features new feedback:

<Tamalpais Valley> has new feedback.


 * Trigger: New feedback 'featured' on page you started (marked as helpful or useful)
 * Main Link: Feedback page for your page (showing new post on top of 'featured' list)
 * Payload: Show comment from featured post
 * Frequency: Medium
 * Priority: Medium (neutral)
 * Category: Feedback


 * Flyout Summary: [icon] <Tamalpais Valley> has new feedback.
 * Archive Summary: [icon] <Tamalpais Valley> has new feedback by anonymous user 76.181.147.57.
 * Email Subject 1: A page you started has new feedback [generic]
 * Email Subject 2: <Tamalpais Valley> has new feedback [specific]
 * Email Summary: <Tamalpais Valley> has new feedback by anonymous Wikipedia user 76.181.147.57.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner. For notifications that are about a page, such as this one, it would seem helpful to include the page name and the user name in the email subject line 2. However, this could be an issue for long article titles, which may need to be truncated after 20? characters, so the subject line can be understood. But if this is too complex for the first release, we could start with just the generic subject line 1.

Negative
This group includes notifications that can make new users feel bad about participating on Wikipedia. In general, we recommend that these notifications only be shown if the user 'opts-in' in their preferences.

Started Page - Marked for deletion
This notification occurs when a page you started is marked as reviewed and also nominated for deletion, including one or more deletion tags (AFD / CSD / PROD):

<Government of Kazakhstan Airline> was reviewed and marked for deletion by <Jetstreamer>, who started. "This article may not be suitable for inclusion in Wikipedia, according to Wikipedia's policies and guidelines."


 * Trigger: Editor review (or patrol) of page you started -- with deletion tags
 * Main Link: Your page (with templates)
 * Payload: Tags
 * Frequency: Low
 * Priority: Medium (negative, but important)
 * Category: Page review


 * Flyout Summary: [icon] <Government of Kazakhstan Airline> was marked for deletion by Tchay. Tags: ...
 * Archive Summary: [icon] <Transcarpathian Art Institute> was reviewed and marked for deletion by <Jorm>. Tags: ...
 * Email Subject 1: A page you started was marked for deletion [generic]
 * Email Subject 2: Angry Birds was marked for deletion by TheHelpfulOne [specific]
 * Email Summary: <Angry Birds> was marked for deletion by Wikipedia user QatarStarsLeague. Tags: ...

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Note that there are about a dozen possible tags that may need to be supported for this notification type.

Edit reverted
This notification occurs when your edit was just reverted (undone or rolled back):

Your edit on <Hurricane Sandy> was by <Knowledgekid87>.

Your edit on <Hurricane Sandy> has been reverted by <NewsAndEventsGuy>. "Undid revision 521530151: With what other sentence in the lead is it redundant?"


 * Trigger: Editor reversion (click 'Undo', 'Rollback', manual or automated)
 * Main Link: Diff page (shows difference between your edit and theirs)
 * Payload: Show 'Edit summary'
 * Frequency: High
 * Priority: Low (negative, opt-in)
 * Category: Edit revert


 * Flyout Summary: [icon] Your edit on <Government of Kazakhstan Airline> was reverted by Knowledgekid87.
 * Archive Summary: [icon] <Transcarpathian Art Institute> was reverted by <Eloquence>.
 * Email Subject 1: Your edit was reverted [generic]
 * Email Subject 2: Your edit on Angry Birds was reverted by NewsAndEventsGuy [specific]
 * Email Summary: Your edit on <Angry Birds> was reverted by Wikipedia user NewsAndEventsGuy.

If needed, we could use the generic email subject 1 above for the first release, to reduce complexity and deploy sooner.

Infrequent Notifications
This group includes notifications that are not likely to happen often for new users -- and therefore could be held back for future releases, even though they can be very positive.

Started Page - Rated
This notification occurs when a page you started receives a Good article rating:

<Angry Birds> was rated as <"Good">.


 * Trigger: Editor adds category to page you started
 * Main Link: Your page (or category page?)
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Recognition

Started Page - Featured
This notification occurs when a page you started is linked from another page:

<Thomas Percy> was just.


 * Trigger: Editor adds category to page you started
 * Main Link: Your page (or category page?)
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Recognition

Started Page - Today's Featured Article
This notification occurs when a page you started is Today's Featured Article:

<Nancy Drew> is <Today's Featured Article>.


 * Trigger: Not sure
 * Main Link: Your page (or Main Page?)
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Recognition

Welcome Page
This notification occurs when a user creates a new account:

Welcome to Wikipedia


 * Trigger: Account creation
 * Main Link: Your profile
 * Payload: None
 * Frequency: Low
 * Priority: Very High (positive)
 * Category: Milestone

Under investigation
This group includes notifications that show potential for engaging users, but that require more research to determine their feasibility.

Answer to your question
This notification occurs when your question is answered (in Help / Teahouse/ Moodbar).

<Kaldari> answered a question you asked in <Teahouse>.


 * Trigger: User edit (in section you started on helpdesk page)
 * Main Link: Helpdesk page (to edited section, if possible)
 * Payload: Show 'Edit summary' (or 'answer first line'?)
 * Frequency: Low

Activity for your recent edit
This notification occurs when other editors contribute to a page you recently edited (and your edit survived).

5 editors contributed to <Porcelain money> since you last edit.


 * Trigger: User edits (with no revert)
 * Main Link: Article page
 * Payload: None
 * Frequency: High

Check out your watchlist
This notification occurs when your watchlist has new activity that relates to you.

<Your watchlist> has 15 new events which may interest you.


 * Trigger: Watchlist events (if you haven't visited watchlist yet)
 * Main Link: Watchlist page
 * Payload: None
 * Frequency: Low (if only shown the first time)

Research Goals
Here are our goals for collecting and analyzing data for the Echo notifications project.

We want to measure how effective Echo is in helping users:
 * learn about activity related to them
 * take action on notifications
 * participate on Wikipedia

We plan to answer these questions through a variety of measurements, some of which would be integrated in the Echo tools. Here are some examples of measurements we could take on, for discussion purposes. After an initial feasibility study and prioritization session, we will identify a few key metrics that we think are required and practical for our first en-wiki deployment.

Target Users
Our initial focus is on new users, but we may want to test the tool with other users as well. Here are the activation options we are considering for testing on the English Wikipedia in early 2013:
 * All users (compare new users vs. active users)
 * New users only (after cut-off date, à la Moodbar)
 * Split test (control group vs. Echo group)

Volume Metrics
Note that some of the breakdowns proposed below (e.g. new vs. active user, or source breakdown) are useful across other types of metrics (e.g. posts vs. views vs. clicks). For example, it would be helpful to have a matrix table showing how many posts, views and clicks were generated by each user group across a range of sources.

Posts
How many notifications are being posted to users in a given week? (but not necessarily viewed)
 * for new users
 * for active users
 * for very active users

Note:
 * Rationale for this request: determine whether specific classes of users receive too many notifications

Views
How many notifications are actually viewed in a given week?
 * on the flyout
 * on the archive page(s)
 * via email (plain text vs. HTML)

Notes:
 * We expect to use EventLogger to track these views
 * We would need to discuss building view tracking into our html email template and a server end to record views
 * Potential privacy concerns for HTML tracking, but OTRS has support for email impressions
 * For users with many pages of archive, how many click past the first page?

Clicks
How many notifications are being clicked on in a given week?
 * interactive notifications (e.g.: talk page messages, user mentions)
 * positive notifications (e.g.: wiki love, page link, promoted feedback)
 * neutral notifications (e.g.: page reviews, user mentions)
 * negative notifications (e.g.: edit reverts, page deletions)

Notes:
 * Break out clickthrough stats by source (flyout/archive/text-email/html-email)

Email preferences

 * How many emails are being sent in a week to new/active/very active users?
 * How many users unsubscribed? switched to daily? weekly?
 * For users who disabled or reduced their email settings, how many emails had been sent to them in the past day/week?

Notes:
 * Control for users with authenticated email addresses

Users
How many unique users are clicking on their notifications:
 * every day
 * once a week
 * once a month
 * rarely or never

Actions
How many users go on to take these follow-up actions?
 * make an article edit (ns0) -- this is the most important action
 * start a page
 * post a message
 * post feedback
 * upload a file
 * other contributions
 * do nothing

Notes:
 * We need to define what is the scope of a "follow up" action.
 * Since it is hard to determine which actions take place as a direct result of a notification, it might make sense to compare a group of users who are getting Echo notifications with a control group without Echo, then track the overall number of edits, new pages and other contributions from both groups over a period of time.
 * This would let us determine whether or not Echo is helping engage users who receive notifications more than other users.

Usage
Which notification types are used the most? the least?
 * Welcome
 * Talk page message
 * Started page message
 * Wikilove
 * etc.

Notes:
 * The purpose of this request is to determine whether any of these notifications should be eliminated due to insufficient usage
 * To normalize the data, we may want to focus on the ratio of clickthrough versus frequency (e.g.: measure the number of clicks divided by number of notifications)

Productivity
Which notification types appear to be most productive in terms of follow-up actions?
 * Welcome
 * Talk page message
 * Started page message
 * Wikilove
 * etc.

Usefulness
How useful are notifications to users overall? to new vs. experienced users?
 * Very useful
 * Useful
 * Neutral
 * Not very useful
 * Not useful at all

Notes:
 * This measurement of customer satisfaction could be based on a simple survey shown to users for a short period of time.

General dashboards
We would like some live dashboards showing aggregated data above, dynamically-generated.

Specific dashboards
We would like some specific dashboards for each notification type, so that we (or developers using our API) can track their effectiveness.

User dashboards
We would like some individual user dashboards, so we can observe how typical users use this tool. It would also be useful to look at an average by user type as well (e.g. new user vs. active user average).