Wikimedia Apps/Team/Release process

This page documents the release process for the Mobile Apps Team.

Android Wikipedia app

Alpha releases

Alpha apks are built automatically after patches get merged to master every 15 minutes and are available at android-builds.wmflabs.org.

Production releases

Production releases are promotions from beta by the product owner.

1. Check out the last beta and push the prod tags.

   git clone https://gerrit.wikimedia.org/r/p/apps/android/wikipedia
   cd wikipedia
   
   declare beta="$(git tag -l beta/\*|tail -n1)"
   declare r="${beta//beta/r}"
   git tag "$r" "$beta"
   echo "Pushing \"$r\"..."
   read -p "Press <enter> to continue, <ctrl-c> to abort: " &&
   git push origin "$r"
   

2. Promote the Play Store alpha build to production and submit the Appstore build.

Beta release

1. Announce on #wikimedia-mobile^connect that a new beta build is in progress and that merges should be held.
2. Prepare the workspace.

   git clone https://gerrit.wikimedia.org/r/p/apps/android/wikipedia # TODO: use Diffusion when writable.
   cd wikipedia
   

3. Increment the patch version code. Consider updating the minor number.

   scripts/bump-version-code.py
   

4. Push the version increment patch.

   git push https://gerrit.wikimedia.org/r/apps/android/wikipedia HEAD:refs/for/master
   

5. +2 the patch.
6. Set the environment variables.

   # TODO: allow date to be injected into make-release names.
   APP_DATE="$(date +%F)"
   # TODO: don't hardcode 2.5.
   APP_VER="2.5.$(sed -rn '/\s*versionCode\s+([0-9]+)\s*/ { s%%\1%p; q }' app/build.gradle)"
   APP_NAME="$APP_VER-$APP_DATE"
   RELEASE_DIR="wikipedia-$APP_NAME"
   
   echo "Testing $APP_NAME"
   

7. For hotfix releases:

   git checkout r/OLD_RELEASE
   git cherry-pick ...
   git cherry-pick ...
   git cherry-pick ...
   git cherry-pick BUMP_VERSION_COMMIT
   

8. Start the release notes Etherpad cooking.

   git log --pretty=format:"%h | %cr | %s" --abbrev-commit --no-merges `git tag --list beta/* | tail -1`.. | grep -Ev 'Hygiene:|Localisation updates from https://translatewiki.net.'
   
   echo 'Volunteer contributor patches:'
   git log --pretty=format:"%h | %an | %cr | %s" --abbrev-commit --no-merges `git tag --list beta/* | tail -1`..|grep -Evi 'Dmitry Brant|Stephen Niedzielski|Michael Holloway|bearND|Translation updater bot'
   

9. Add a release card to the current sprint.
10. Launch a few emulators (with generous disk space and low resolution) for APIs 17, 19, and 21 and connect all physical devices including a Fire OS device and the latest API. Note: CI tests on API 15. For physical devices, enable the "stay awake" developer option. On one device, enable the "don't keep activities" developer option. TODO: set this in script.
11. Verify the number of physical and virtual devices connected is expected.

    adb devices
    

12. Confirm the version patch has merged and start an alpha build.
13. Build the beta, production, and partner flavors.

    ~/Downloads/all-apks.sh
    

    This script is confidential because some partners wish to maintain anonymity. However, it looks something like this:

    #!/usr/bin/env bash
    
    set -euo pipefail
    
    make-release() { scripts/make-release.py "$@"; }
    
    make-release --beta
    make-release --prod
    make-release --amazon
    
    # TODO: version script and pass the partners as a string.
    for channel in foo bar baz boo bug oog; do
      make-release --channel "$channel"
    done
    

14. Install beta and prod apks on all devices.

    declare devices=( $(adb devices|sed -nr '2,$ s_([^\t ]+).*_\1_p') )
    echo "${#devices[@]} ADB devices detected."
    
    # $1 apk
    apk_to_pkg() {
      declare apk="$1"
      aapt d badging "$apk"|
      sed -rn "0,/package: name='([^']+)'.*/ { s%%\1%p; q }"
    }
    
    # $1 apk
    apk_to_activity() {
      declare apk="$1"
      aapt d badging "$apk"|
      # TODO: only need first match.
      sed -rn "s%launchable-activity: name='([^']+)'.*%\1%p"
    }
    
    cd releases
    
    # TODO: add adb-setup to test like CI.
    # TODO: error on failure instead of manually checking results.
    for i in wikipedia-*-r-*.apk wikipedia-*-beta-*.apk; do
      for serialno in "${devices[@]}"; do
        echo "Installing $i on $serialno..."
        adb -s "$serialno" install -r "$i"
        declare pkg="$(apk_to_pkg "$i")"
        adb -s "$serialno" shell "am start -R5 -W -a android.intent.action.MAIN -c android.intent.category.LAUNCHER $pkg/$(apk_to_activity "$i")"
      done
    done
    
    cd ..
    

15. Tap the "keep this build forever" button in CI.
16. Do some light manual testing, including search, saved pages, reading lists, history, nearby, beta-only and prod-only features, Content Service-only and MediaWiki-only features, light and dark mode, upgrade from a previous version, and new features and churn areas identified when generating release notes.
17. Upload the -r- to the Play Store production alpha. TODO: automate.
18. Push the tags.

    scripts/make-release.py --beta --push
    

19. Preserve the apks on releases.wikimedia.org.

    # TODO: use rsync.
    scp releases/wikipedia-*-beta* bromine:/srv/org/wikimedia/releases/mobile/android/wikipedia/betas/
    ssh bromine ls -ltc /srv/org/wikimedia/releases/mobile/android/wikipedia/betas/|grep $APP_VER
    
    scp releases/wikipedia-*-releasesprod-* bromine:/srv/org/wikimedia/releases/mobile/android/wikipedia/stable/
    ssh bromine ls -ltc /srv/org/wikimedia/releases/mobile/android/wikipedia/stable/|grep $APP_VER
    

20. Upload the releases to Google Drive too.

    mv releases $RELEASE_DIR
    time drive upload -p <folder id> -f $RELEASE_DIR
    # TODO: upload apk for TSG testing.
    

21. Verify the files were uploaded and move the old files to the "old" folder.
22. Review the alpha pre-launch test results.
23. Upload the -beta- to the Play Store beta production. TODO: automate.
24. Upload the -amazon- to the Appstore upcoming. TODO: automate.
25. Add the release to Mobile/Release_history under the Android section's "Release notes</translate>".

    echo "|-
    | [https://play.google.com/store/apps/details?id=org.wikipedia Android] || $APP_VER-r-$APP_DATE || [https://releases.wikimedia.org/mobile/android/wikipedia/stable/wikipedia-$APP_VER-releasesprod-$APP_DATE.apk] || $APP_DATE || $(md5sum $RELEASE_DIR/wikipedia-*-releasesprod-*.apk|sed -r 's%(\W).*%\1%')||
    <translate>Release candidate alpha</translate>"
    # TODO: automate.
    

    With the following commit message:

    echo "Add Wikipedia Android app v$APP_VER-releasesprod-$APP_DATE"
    

26. Send email to mobile-l, cc android, and bcc volunteer contributors. Don't forget to call out volunteers! TODO: templatize.
27. Send email to TSG. TODO: templatize.

iOS

See the following:

• https://etherpad.wikimedia.org/p/iosappcadencenojoke (1-April-2015)
• https://etherpad.wikimedia.org/p/qaoniosrocks (2-April-2015)
• https://etherpad.wikimedia.org/p/iosreleasing (8-April-2015)

Cards

Effective mid-sprint 54 on iOS, Phabricator cards on iOS will go through Code Review, QA Signoff, and Design Sign Off before they go to the Ready for Review (Product Manager signoff). We'll tweak the process as necessary. Important points about this:

• The Description field should contain a link to the design card. The design card should contain mocks, expected user interaction behavior, and any finer points that may escape the attention of engineers (e.g., gradients, font faces, transition durations, etc.). It should address phone and tablet form factor, as well as portrait and landscape mode.
  • The spec should be noted to reading-wmf one week before a sprint starts to allow feedback
• The Product Manager and Designer will ensure the Description field contains sufficient Acceptance Criteria for QA to know what to test (if not already captured on the Design card), and software engineers should update if there are specifics relating to the patch associated with a card.
• The primary code author must test on iOS 6 and iOS 8 (on 1 form factor at least, both phone and tablet if UI related). Code reviewers must test on at least one device. The QA analyst must test on iOS 6, iOS 7 (simulator, less emphasis), and iOS 8 on a phone form factor, and must test on at least one version of iOS for the tablet form factor (ideally, both iOS 8 and iOS 7, though, for tablet form factor instead of just one iOS version). Recommendation: test daily.
  • Alpha builds will be available each weekday morning, so QA should be able to review cards in the QA column at that time. If a crash occurs, QA should start the app back up and submit the crash via HockeyApp.
• Design to sync with QA twice per week on any cards in its column for sign off. One approach: test alpha Wednesday and Friday of week 1, Tuesday and Thursday of Week 2 for all cards in Design column. If a crash occurs, Design should start the app back up and submit the crash via HockeyApp.
• The developer working the card should shepherd it through the columns (e.g., prompt QA or Design if necessary). Engineers, QA, Design, Product Manager, and Scrummaster should be online on IRC during normal business hours in case they need to be nudged about a card.
• Work product may be rejected and moved back to To Do, with a Comment explaining why, if the implementation did not match the specification. Otherwise, new cards (Tasks) should be filed if follow-on work is required, and the work will be prioritized for later implementation (potentially in the current sprint, but potentially in a later sprint); bug reporting guidelines should be used for such follow-on work as appropriate.
• If sign off is not required from a particular stakeholder (e.g., Design not needed for data layer testing), the Description card should indicate as much, and the card should be dragged to the appropriate column.
• The engineer should indicate low level technical testing steps as appropriate to the technical depth of the card.
• If necessary, note whether the card is high risk.
• Features that we know can't be used by the end of a sprint should not submitted for TestFlight or App Store.
• Update the iOS Test Cases for regression testing as necessary.

Philosophy

Release every two weeks. Keep it simple. Give a solid reason if blocking a release. But it's okay if it has to be blocked. This is how we roll.

Feature cutoff is Friday afternoon 3 PM San Francisco on week 2 of the sprint. Cards that don't make it by then get moved to the next sprint or backlogged. Enjoy your weekends!

Release card

There will be a release checklist card added to every sprint board. This should be linked into the roadmap board.

Week 1 of iteration

• Monday morning: Tech Lead engages Specialists Guild, informing them that Wikipedia Mobile will be released via TestFlight the following day and that their testing results are requested to be returned by Wednesday no later than noon.

• Monday afternoon: 3 PM San Francisco last minute bugfix code cutoff for prior sprint's work. Ideally, there won't actually be any new code on Monday pertaining to the prior sprint.

• Tuesday morning, 9:30 AM San Francisco time: whatever's merged into the release branch goes into both Wikipedia Beta for TestFlight external testing and Wikipedia Mobile (stable) for TestFlight internal upgrade smoketesting as the release candidate (instructions). The corollary of this is the engineers need to be judicious about not polluting the code; there are many ways to deal with this.

• Thursday afternoon at 3 PM San Francisco time there hasn't been serious badness identified (iOS engineering to report all crashes and user reported feedback), we then submit this release candidate to Apple.

Week 2 of iteration

Thursday morning: Product Manager gives final go/no-go for submitting for releasing.

Release

The app should be submitted during early business hours Monday through Thursday in San Francisco, but not Friday, Saturday, or Sunday. Typically this actually should happen the first week of a sprint. Note that if Comms is needed for a big announcement for a given release, they need a heads up well ahead of time, with timely updates as the ship date approaches (the final 5 business days leading up to big announcements are pretty critical with history as a guide). Big splashes don't happen every sprint (more likely, just quarterly), and for most sprints, social media only announcements with less time-pegged blog posts (if any blog post) is fine.

If we hit blockers

Notify Comms, update the release card, etc.

Version Bumps

Note that TestFlight version bumps on the x.y (major.minor) part of the x.y.z version require formal Apple review before they can become available to external testers. Therefore, if a major.minor bump is needed, the TestFlight toggle needs to be flipped in iTunes Connect to get that in the queue with sufficient lead time so that Tuesday cutpoints aren't missed. Know what you're doing before you toggle switches. You don't want to end up effectively disabling in-flight testing on release candidates.

Unit tests

Broken unit tests on the build server constitute a broken build and must be fixed to make work product shippable.

Open questions

• Branching strategy to handle rejected cards. Realistically, if rejected cards are treated within the same sprint they're worked, no special branching is required. Daily (QA) and twice weekly (Design) feedback should help to avoid serious problems occurring too late in the sprint.
• Branching strategy for the week 1 or week 2 swarms when they're required. Presently, the thought is to just branch from the corresponding tag and do all work on that branch, merging such fixes upstream to master as well.
• The format of what's required in the Description field of cards in Phabricator and what constitutes a sufficient Design card (Trello) isn't fully defined, although most of it is specified above. This said, a checklist could potentially help reduce ambiguity.
• Is a "Spec" column needed on the main iOS board? It would be before the Next Sprint column to reflect when a design specification is still required. Likewise, is an Estimation and a Released column needed in the main iOS board? This way we have a clear liner progression on the main board from Needs Triage to Released.
• Is full regression testing (TSG) required in case submission for app store review was pushed to week 2 of the sprint? Probably not, as the entire team should be swarming on the issues that caused the delay.

See also

• Release criteria