VERP

Adding proper email bounce handling to MediaWiki (with VERP)

 * Public URL: (https://www.mediawiki.org/wiki/VERP)
 * Bugzilla report: (https://bugzilla.wikimedia.org/show_bug.cgi?id=46640 )
 * Announcement:(http://lists.wikimedia.org/pipermail/wikitech-l/2014-March/074911.html].

Name and contact information

 * Name: Tony Thomas
 * Email: 01tonythomas@gmail.com
 * IRC or IM networks/handle(s): tonythomas
 * Location: Kerala, India
 * Timezone: Kolkata,INDIA, UTC+5:30
 * Typical working hours: 5pm to 12:30am (workdays) 10:00am to 9:30pm (weekends)

Project Summary
It's likely that many Wikipedia accounts have a validated email address that once worked but is out of date. Wikipedia do not currently unsubscribe users who trigger multiple non-transient failures and some addresses might be 10+ years old. The wiki should not keep sending email that is just going to bounce. It's a waste of resources and might trigger spam heuristics. Two API calls need to be implemented: For the second call, authentication will be needed so fake bounces are not a DoS vector or a mechanism for hiding password reset requests. The reason for the threshold is that some failure scenarios will resolve themselves, eg mailbox over quota, so we don't want to react to one bounce. We want a history of consecutive mails bouncing. There would be a Mediawiki development component to this task to build the API, to add VERP request calls wherever email is sent, and an Ops component to route VERP bounces to a script (taking the mail as stdin, and optionally e.g. the e-mail address as arguments), which can then call the (authenticated) MediaWiki API method to remove the mail address. Since its the time MediaWiki mail infrastructure is being moved to new Data Center, this is the right time to implement VERP.
 * 1) One to generate a VERP address to use when sending mail from MediaWiki.
 * 2) One that records a non-transient failure.  That API call would record the current incident and if there had been some threshold level met, eg at least 3 bounces with the oldest at least 7 days ago, then it would un-confirm the user's address so mail will stop going to it.

VERP stands for Variable Envelope Return Path, and on implementation alters the default envelope sender. For eg: if an email needs to be send to bob@example.com, VERP alters the default envelope sender from : wiki@wikimedia.org to a prefix/delim/hash:  [bob][-][mdfkdjw6R4xGdiflfdfkQ]@wikimedia.org, so that the bounce can be used more effectively. The API would record the return address of the bounce and deduce that a mail to bob have failed. On consecutive failures, say at least 3 bounces with the oldest at least 7 days ago, the second API un-confirms the user's address.

The return path address needs to be a prefix/delim/hash as to avoid fake bounces DoSing a user. The VERP address will generally look like this : bounce-{$key}@wikimedia.org The prefix /^bounce-/ is used by the incoming MTA as a hook to route messages to the bounce processor, and $key is used by the bounce processor to figure out which wiki user is having delivery issues. An attacker needs to be prevented from spoofing bounce messages and causing mass unsubscribes. This can be accomplished by making $key secret, and not a simple hash that can be reversed or guessed. Generating an HMAC, with a secret key, over a string containing the user's email address, timestamp, and the list name will be the best option as per security experts in MediaWiki. HMAC can be generated by one of PHP's built in function.
 * Possible mentors: Jeff Green, Kunal Mehta

Problem Background
When an email is sent, on the Wiki web server a message is injected to the local MTA in a shell call by the user the MediaWiki web server daemon runs under. MediaWiki uses the config variable $wgPasswordSender to set the envelope sender, and all messages are sent as the user (for example 'wiki@wikimedia.org'). In WMF's environment, the webserver's MTA is configured to route all messages through the organization's main mail server, which relays them to the destination/remote server as determined from DNS MX records. There are many points where the delivery can fail, for example: Each case can result in the mailserver currently handling the transaction to originate a bounce message. So a bounce can originate within the local system (i.e. the WMF environment) or the remote system (the recipient's environment). Bounce messages generally go back to the envelope sender. Currently, in the case of WMF's system, bounces coming back to wiki@wikimedia.org are sent to /dev/null.
 * DNS lookup failure (Permanent failure)
 * Network failure (Temporary failure)
 * Remote server could be overloaded (Temporary failure)
 * Remote server might blacklisted wikimedia.org or wiki@wikimedia.org (Temporary failure)
 * Remote server could say example@gmail.com is a bad address (Permanent failure)
 * Remote server could say example@gmail.com is over quota (Temporary failure)

Deliverables
Since its time the WMF is moving its servers to a new data center and the mail infrastructure is being rebuilt, this is the right time to implement the functionality. The final results should be :
 * All emails for users of WMF-hosted wikis should have their default envelope sender changed from wiki@wikimedia.org to a VERP generated envelope sender as (prefix/delim/hash) say : [bob][_][mdfkdjw6R4xGdiflfdfkQ]@wikimedia.org
 * If the mail delivery fails due to any of the problem discussed above, a return mail should reach WMF mail servers with the receipient [bob][_][mdfkdjw6R4xGdiflfdfkQ]@wikimedia.org, and an API running there should record the failure and check for the past history of bounces of for this user from a database, and unconfirm the user if threshold level met.
 * The VERP generated recipient address will be the output of an HMAC with a secret key, over a string containing the user's email address, timestamp, and the list name.

Workflow
As stated, my project involves work on the System Ops component to alter the behavior of McHenry, and a MediaWiki development component to build the API's. I had been discussing with my mentors (Jeff Green and Legoktm) and came out with an environment model to automate the whole setup using multiple Virtual Boxes. The setup involves a Linux machine ( Box1 ) hosting MediaWiki communicating to a second Linux server ( Box2 ) with postfix, bind9 and mutt installed acting as a router to the external world. The Box2 acts as a DNS server for the internal network and have NAT enabled so that, it can connect to the external world, and is configured to intercept any mails/packets passing its way and redirect it to /var/www/mail/root folder. This make sure that all emails sent from Box1 reaches safely into Box2 for inspection. The MediaWiki instance is made accessible from my host by a defined ip. We are successful in getting this stage done, now I will have to move it to the Wikimedia Labs instance to include collaboration with my mentors and the community. Later API for Verp can be implemented in Box1, and bounce calls originate on Box2 to implement the feature, which can be implemented in the WMF side.

Participation
Personally, I am strict with the idea of sharing. It's always - Think free, code better. This has always helped me to get well with the WikiMedia community. I find time to write about something new learnt in my blog, and it will be the central focus of all progress reports. All source code I write will be published daily to my Github repo and Gerrit to make sure of collaboration. I try always to stay live in IRC, and am regular in replying to emails, allowing me to keep pace with the community. Testing and documentation will be added to the Wikitech Mail page. I had always found IRC's better than hangouts in getting things done, and always will utilize #mediawiki and #wikimedia-dev to the maximum.

About Me
I am a 19 year old Computer Science Engineering student from Amrita Vishwa Vidyapeetham, Kerala, India. I am an active member of the FOSS community here - FOSS@Amrita. The FOSS club helps me work with code even late night in my college lab. I am a consistent user of Linux  for the past three years. The feeling of Open Source is so compelling, you can never quit contributing to them. I found the Wikimedia Community one of them. My first contribution to Open Source was a bug fix to MediaWiki almost six months before. Since then, I was working with the codebase, fixing and looking for errors, and creating new. I was able to help and mentor many of my FOSS club mates to contribute to MediaWiki, you can see the full list here. Along with academics I find myself time to work in the lab from 5:00 pm to 11:00 am on all working days and 10:00 am to 11:00 pm on weekends.

I have chosen as adding functionality to the Email Component as my project as it involves both the server side and MediaWiki side development. As one of my mentors mentioned it, I am sure its going to be a fun project, and helps me implement all the Networking lessons I learnt when I did a MCITP networking course before. The main aim is of course studying new things, understanding huge code bases, but this involves a hardware element too, which excites me. Coming from a remote village in Kerala I think this would a boost for me to spread Open Source in a much removed society, who still are ignorant about collaboration and thinking freely. You can check my blog here - Through my Pages.

Past open source experience
I have contributed to the Wikimedia Project by various bug fixes in MediaWiki Core, Extensions and Browsertests.
 * Gerrit Changes: Gerrit owner :01tonythomas
 * GitHub profile: tonythomas01
 * Co-authored an app for Firefox OS Marketplace - Daily Wallet