Wikimedia Apps/Team/iOS/Engineering Pre-release Checklist
 | This page is under construction Please help review and edit this page. |
Every iOS release goes through a pre-release checklist, based on a Phabricator template ticket here. This wiki goes into further detail on what all engineering needs to do for this. Results should be pasted as a comment on the release-specific pre-release phabricator ticket as well as here if it would be useful to glance for an overall trend.
Check for retain cycles[edit]
Go over new flows in Instruments to confirm expected classes are properly deallocated.
Run static analyzer and runtime checks[edit]
- Fix any issues surfaced from Product > Analyze. You can ignore any problems within third party libraries.
- In the Wikipedia scheme, under Run > Diagnostics, check off Address sanitizer, detect use of stack after return, Thread sanitizer, and Undefined Behavior sanitizer. Run the app with these various settings and fix any runtime issues that crop up.
- Main thread checker is selected by default here, so no need to explicitly test for this.
- Run the app again with the Core Data thread checker on by adding
-com.apple.CoreData.ConcurrencyDebug 1as a launch argument to the scheme. Run the app (including performing a migration from the last release) and fix any issues that crop up.
Record binary size in TestFlight[edit]
| Device | OS | Release | Size | % change |
|---|---|---|---|---|
| ? | ? | 6.6.0 (1742) | 33.1MB | |
| iPad Pro (11-inch) | iOS 14.4 | 6.8.0 (1803) | 35.4MB | ↑07% |
| iPhone 7 | iOS 14.4.2 | 6.8.0 (1803) | 35.4MB | ↑07% |
| iPad Pro (11-inch) | iOS 14.8.1 | 6.8.2 (1867) | 33.5MB | ↓05% |
| iPhone 7 | iOS 14.8.1 | 6.8.2 (1867) | 33.5MB | ↓05% |
| iPad Pro (11-inch) | iOS 15.4.1 | 6.9.0 (1921) | 38.4 MB | ↑15% |
| iPhone 7 | iOS 15.4.1 | 6.9.0 (1921) | 38.3 MB | ↑14% |
Record launch time[edit]
Do these steps for at least two physical devices and record the results.
- Run the "App Launch" instrument in instruments. Add up the number of ms it takes to perform each purple process and record it below. These are the pre-main times.
- Do this for a fresh install and for upgrading from an old app with many saved articles (I choose 100 over a variety of languages).
| Device | OS | Release | Type | pre-main time | % change |
|---|---|---|---|---|---|
| iPhone 7 | 13.4.1 | 6.6.0 | Fresh install | 318.39ms | |
| iPhone 7 | 13.4.1 | 6.6.0 | Upgrade | 326.69ms | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 | Fresh install | 328.94ms | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 | Upgrade | 405.57ms | |
| iPhone 7 | 14.4.2 | 6.8.0 | Fresh install | 261.58ms | ↓18% |
| iPhone 7 | 14.4.2 | 6.8.0 | Upgrade | 229.45ms | ↓30% |
| iPad Pro 11-inch | 14.4 | 6.8.0 | Fresh install | 246.05ms | ↓25% |
| iPad Pro 11-inch | 14.4 | 6.8.0 | Upgrade | 248.27ms | ↓39% |
| iPhone 7 | 14.8.1 | 6.8.2 | Fresh install | 93.73ms | ↓64% |
| iPhone 7 | 14.8.1 | 6.8.2 | Upgrade | 83.00ms | ↓64% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | Fresh install | 242.78ms | ↓01% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | Upgrade | 254.39ms | ↑02% |
| iPhone 7 | 15.4.1 | 6.9.0 | Fresh install | 231.43ms | ↑147% |
| iPhone 7 | 15.4.1 | 6.9.0 | Upgrade | 221.39ms | ↑168% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | Fresh install | 194.73ms | ↓20% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | Upgrade | 177.32ms | ↓30% |
Run manual unit tests & stub performance tests[edit]
Currently we have some manual and performance tests that are not set up to work with our build server / CI process. Until then, it's good to run these manually pre-release as a sanity check. These tests are in ArticleCacheReadingManualTests,swift, ReadingListManualPerformanceTests.swift, ArticleManualPerformanceTests.swift, and TalkPageManualPerformanceTests.swift. Run these on at least two devices from the Performance Testing scheme. Confirm ArticleCacheReadingManualTests run at all (will need to be on a device with no connection), and record the average run times of the performance tests. Note ArticleManualPerformanceTests.swift stub out the network call with Nocilla, so it is a true native-only performance test.
| Device | OS | Release | Test name | average time | % change |
|---|---|---|---|---|---|
| iPhone 7 | 14.4.2 | 6.6.0 (1742) | testArticlePeekPreviewControllerDisplayTime | 0.085s | |
| iPhone 7 | 14.4.2 | 6.6.0 (1742) | testArticleSetupTime | 0.452s | |
| iPhone 7 | 14.4.2 | 6.6.0 (1742) | testContextMenuConfigTime | 0.120s | |
| iPad Pro 11-inch | 14.4 | 6.6.0 (1742) | testArticlePeekPreviewControllerDisplayTime | 0.061s | |
| iPad Pro 11-inch | 14.4 | 6.6.0 (1742) | testArticleSetupTime | 0.351s | |
| iPad Pro 11-inch | 14.4 | 6.6.0 (1742) | testContextMenuConfigTime | 0.093s | |
| iPhone 7 | 14.4.2 | 6.8.0 | testArticlePeekPreviewControllerDisplayTime | 0.051s | ↓40% |
| iPhone 7 | 14.4.2 | 6.8.0 | testArticleSetupTime | 0.501s | ↑11% |
| iPhone 7 | 14.4.2 | 6.8.0 | testContextMenuConfigTime | 0.106s | ↓12% |
| iPad Pro (11-inch) | 14.4 | 6.8.0 | testArticlePeekPreviewControllerDisplayTime | 0.039s | ↓36% |
| iPad Pro (11-inch) | 14.4 | 6.8.0 | testArticleSetupTime | 0.309s | ↓12% |
| iPad Pro (11-inch) | 14.4 | 6.8.0 | testContextMenuConfigTime | 0.080s | ↓14% |
| iPhone 7 | 14.8.1 | 6.8.2 | testArticlePeekPreviewControllerDisplayTime | 0.059 | ↑15% |
| iPhone 7 | 14.8.1 | 6.8.2 | testArticleSetupTime | 0.456s | ↓9% |
| iPhone 7 | 14.8.1 | 6.8.2 | testContextMenuConfigTime | 0.150s | ↑41% |
| iPad Pro (11-inch) | 14.8.1 | 6.8.2 | testArticlePeekPreviewControllerDisplayTime | 0.050s | ↑28% |
| iPad Pro (11-inch) | 14.8.1 | 6.8.2 | testArticleSetupTime | 0.329s | ↑6% |
| iPad Pro (11-inch) | 14.8.1 | 6.8.2 | testContextMenuConfigTime | 0.090s | ↑12.5% |
| iPhone 7 | 15.4.1 | 6.9.0 | testArticlePeekPreviewControllerDisplayTime | 0.036s | ↓39% |
| iPhone 7 | 15.4.1 | 6.9.0 | testArticleSetupTime | 0.463s | ↑2% |
| iPhone 7 | 15.4.1 | 6.9.0 | testContextMenuConfigTime | 0.042s | ↓72% |
| iPad Pro (11-inch) | 15.4.1 | 6.9.0 | testArticlePeekPreviewControllerDisplayTime | 0.036s | ↓28% |
| iPad Pro (11-inch) | 15.4.1 | 6.9.0 | testArticleSetupTime | 0.326s | ↓1% |
| iPad Pro (11-inch) | 15.4.1 | 6.9.0 | testContextMenuConfigTime | 0.064s | ↓29% |
Run live tests[edit]
As a last set of performance tests, we are recording the time it takes to display article content via os_log. This is printed out in the console when you run in the Performance Testing scheme. You can also try searching via the Device log in the Console app for:
process:Wikipedia subsystem:org.wikimedia.ios category:articleLoadTime
if you want to find these values without connecting to Xcode. Be sure Action > Include Info Messages is checked.
To reach this log, go to the Explore feed and search for an article. The load time is recorded from just before an article is presented, to the PCS Javascript 'setup' action that is called over the bridge informing native land that initial article loading is complete. For these tests, uncheck the "Debug executable" checkbox in the Run section of Edit Scheme for Performance Testing so that this is closer to a live run. We want to run on at least 2 devices, with a large and a small article, for a fresh install, a cached article load, and a saved article load.
Fresh install[edit]
For each device:
- Fresh install the app.
- Go to iOS Settings > Safari > Clear History and Website Data.
- Search United States on EN Wikipedia. Record time. Record result.
- Repeat steps 1-3 for Puppy Cat on EN Wikipedia. Record result.
| Device | OS | Release | Article | Time | % change |
|---|---|---|---|---|---|
| iPhone 7 | 13.4.1 | 6.6.0 (1742) | United States | 1.14s | |
| iPhone 7 | 13.4.1 | 6.6.0 (1742) | Spinning Newspaper | 0.70s | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 (1742) | United States | 0.61s | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 (1742) | Spinning Newspaper | 0.32s | |
| iPhone 7 | 14.4.2 | 6.8.0 | United States | 1.25s | ↑09% |
| iPhone 7 | 14.4.2 | 6.8.0 | Puppy cat | 0.36s | |
| iPad Pro 11-inch | 14.4 | 6.8.0 | United States | 0.50s | ↓18% |
| iPad Pro 11-inch | 14.4 | 6.8.0 | Puppy cat | 0.45 | |
| iPhone 7 | 14.8.1 | 6.8.2 | United States | 0.80s | ↓36% |
| iPhone 7 | 14.8.1 | 6.8.2 | Puppy cat | 0.31s | ↓13% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | United States | 0.65s | ↑30% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | Puppy cat | 0.24s | ↓47% |
| iPhone 7 | 15.4.1 | 6.9.0 | United States | 1.51s | ↑89% |
| iPhone 7 | 15.4.1 | 6.9.0 | Puppy cat | 0.34s | ↑10% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | United States | 0.73s | ↑12% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | Puppy cat | 0.37s | ↑54% |
Revisit[edit]
For each device, after your last article visit from fresh install (i.e. while it's still cached):
- Revisit article from Search result. Record the average.
| Device | OS | Release | Article | Time | % change |
|---|---|---|---|---|---|
| iPhone 7 | 13.4.1 | 6.6.0 (1742) | United States | 0.77s | |
| iPhone 7 | 13.4.1 | 6.6.0 (1742) | Spinning Newspaper | 0.43s | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 (1742) | United States | 0.43s | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 (1742) | Spinning Newspaper | 0.27s | |
| iPhone 7 | 14.4.2 | 6.8.0 | United States | 0.75s | ↓03% |
| iPhone 7 | 14.4.2 | 6.8.0 | Puppy cat | 0.31 | |
| iPad Pro 11-inch | 14.4 | 6.8.0 | United States | 0.35s | ↓19% |
| iPad Pro 11-inch | 14.4 | 6.8.0 | Puppy cat | 0.24s | |
| iPhone 7 | 14.8.1 | 6.8.2 | United States | 0.57s | ↓24% |
| iPhone 7 | 14.8.1 | 6.8.2 | Puppy cat | 0.26s | ↓16% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | United States | 0.46s | ↑31% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | Puppy cat | 0.24s | 0% |
| iPhone 7 | 15.4.1 | 6.9.0 | United States | 0.91s | ↑60% |
| iPhone 7 | 15.4.1 | 6.9.0 | Puppy cat | 0.31s | ↑19% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | United States | 0.72s | ↑57% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | Puppy cat | 0.31s | ↑29% |
Saved article[edit]
For each device, tap save article icon on Article for your long and short article (eg. United States and Spinning Newspaper). Then for each article:
- Go to App Settings, tap "Clear cached data"
- Go to iOS Settings > Safari > Clear History and Website Data.
- Enter airplane mode. Revisit article via Saved tab. Note result and record.
| Device | OS | Release | Article | Time | % change |
|---|---|---|---|---|---|
| iPhone 7 | 13.4.1 | 6.6.0 (1742) | United States | 0.86s | |
| iPhone 7 | 13.4.1 | 6.6.0 (1742) | Spinning Newspaper | 0.37s | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 (1742) | United States | 0.34s | |
| iPad Pro 11-inch | 13.4.1 | 6.6.0 (1742) | Spinning Newspaper | 0.19s | |
| iPhone 7 | 14.4.2 | 6.8.0 | United States | 0.69s | ↓20% |
| iPhone 7 | 14.4.2 | 6.8.0 | Puppy cat | 0.30s | |
| iPad Pro 11-inch | 14.4 | 6.8.0 | United States | 0.40s | ↑18% |
| iPad Pro 11-inch | 14.4 | 6.8.0 | Puppy cat | 0.24s | |
| iPhone 7 | 14.8.1 | 6.8.2 | United States | 0.55s | ↓20% |
| iPhone 7 | 14.8.1 | 6.8.2 | Puppy cat | 0.22s | ↓26% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | United States | 0.55s | ↑37% |
| iPad Pro 11-inch | 14.8.1 | 6.8.2 | Puppy cat | 0.29s | ↑20% |
| iPhone 7 | 15.4.1 | 6.9.0 | United States | 0.95s | ↑73% |
| iPhone 7 | 15.4.1 | 6.9.0 | Puppy cat | 0.27s | ↑23% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | United States | 0.67s | ↑22% |
| iPad Pro 11-inch | 15.4.1 | 6.9.0 | Puppy cat | 0.43s | ↑48% |