Wikimedia Apps/Team/Android/App hacking
The Wikipedia Android app is completely open-source and welcomes contributions from all.
We welcome volunteer contributions![edit]
Follow the steps below to get started, and feel free to ping any member of the WMF Android dev team (dbrant, cooltey, sharan) on #wikimedia-mobile connect for assistance.
You can find our coding conventions here.
Open bugs and feature requests[edit]
Work items are tracked on our work board. First time app contributors are encouraged to consider an good first task. Unassigned tasks in the "Bug Backlog", "Tech Debt Backlog", and "Product Backlog" columns are generally suitable for development and roughly organized by descending priority. (Please do not work on tasks in other columns like "Open Questions" and "Needs Triage".)
Set up a development environment[edit]
Create a GitHub account[edit]
See https://github.com/.
Add your SSH key to GitHub account[edit]
See Adding a new SSH key to your GitHub account
Install JDK 8[edit]
JDK 8 is needed to build the Android app. To find your current version:
$ javac -version javac 1.8.0_102
The open source OpenJDK may be installed on Debian / Ubuntu by executing sudo apt install openjdk-8-jdk. The closed source Oracle JDK may be installed from here.
Download and install Android Studio[edit]
https://developer.android.com/sdk/index.html
Run the setup wizard. Standard installation is fine. Installation will take a while.
Download the project source[edit]
With SSH credentials (requires GitHub account):
git clone git@github.com:wikimedia/apps-android-wikipedia.git
Or anonymously:
git clone https://github.com/wikimedia/apps-android-wikipedia.git
Open in Android Studio[edit]
Open Android Studio, select "Open an existing Android Studio Project" from the main window, and select the repository folder from the file browser.
Important: If you are prompted about upgrades to components (e.g., Gradle or the Android Gradle plugin) during the initial build, please click "Don't remind me again for this project." We update these components in dedicated patches after a thorough review of the release notes, but they have a way of showing up in unrelated patches if automatic IDE prompting is enabled.
Third-Party Plugins[edit]
Checkstyle-IDEA[edit]
IntelliJ's Checkstyle-IDEA plugin is useful in ensuring that your code will meet the project style guidelines. Install from Android Studio > File > Settings > Plugins, restart to complete the installation, then add the custom configuration file for the project:
Android Studio > File > Settings > Other Settings > click on the '+' in the upper panel and select /gradle/src/checkstyle.xml.
Make sure you activate the new configuration file by checking the checkbox next to it, so our checkstyle rules are enabled and deviations show up as errors in the code editor.
Make changes and submit for review via GitHub[edit]
Before making changes, please create a branch that is associated with a specific task in Phabricator.
git checkout -b Txxxxxx
If the changes are not associated with any task in Phabricator, you can create a branch with a meaningful name of it.
Make and commit your changes, and provide a commit message describing your changes. (Please ensure that the commit message includes line breaks for lines running over 75 characters.) If your work is associated with a specific task in Phabricator, please include, as the last line of the commit message, the following line:
Bug: Txxxxxx (where xxxxxx represents the number of the Phabricator task, e.g., T149500).
After creating your commit, if this is your first of submitting the changes to the branch, you may need to set the upstream to it.
git push --set-upstream origin BRANCHNAME
Once the command is done above, you can simply use the following command when you are in the branch you've been working on.
git push
You'll most likely receive requests to update and resubmit the patch. (It happens to all of us!) Make these changes locally and use git commit to submit your update changes, and make sure the commits are in the same branch.
After submitted a new branch to the GitHub, you can create a Pull Request for other developers to review the changes. For more information, see Creating a pull request
Tip: To avoid CI test failures, it's a good idea to run checkstyle (./gradlew checkstyle) and the unit test suite (./gradlew testDevDebug or ./gradlew tDD) from the project root before pushing to GitHub.
Automated Tests[edit]
The codebase supports two kinds of tests:
- JVM JUnit (preferred, off device, fast, less flaky)
- Android instrumentation (on device)
Command line usage[edit]
- JVM JUnit
./gradlew testDevDebug
- Android instrumentation
./gradlew connectedDevDebugAndroidTest
- Both JVM JUnit and Android instrumentation (including screenshot tests)
./gradlew testAllDevDebug
Tips[edit]
- When running
testAll*, add the--continueflag to run the Android instrumentation tests even if the JVM JUnit tests failed.
Android Studio setup[edit]
Both kinds of tests are supported in Android Studio. You may toggle between them by changing the Test Artifact from View -> Tool Windows -> Build Variants. "Unit Tests" are JVM JUnit tests and "Android Instrumentation Tests" are what it says. A run configuration must be made to prior to executing the tests.
Run configurations[edit]
JVM JUnit[edit]
- Click Run -> Edit Configurations...
- Click Add New Configuration (a large plus icon) and select Android JUnit.
- Name the configuration.
- Change the Test kind to All in the package.
- Set the package to org.wikipedia.
- Ensure the working directory is set to $MODULE_DIR$.
- Set Use classpath of module to
app.
If you cannot run the JUnit test based on the configuration above, please try to remove the .idea/ and .gradle/ folder and then rebuild the project
Android instrumentation[edit]
testAll[edit]
Screenshot tests[edit]
Espresso driven Android UI tests have been created mainly for the purpose of monitoring UI regressions. Learn more about Espresso testing here.
You can find all our Espresso tests at the project folder located at androidTest/java/org/wikipedia/espresso.
Set Up :
1. Device Configuration :
Uncheck the ‘Has Hardware Buttons (Back/Home/Menu). This gets rid of the system navigation bar, allowing the app screen to occupy the entire space.
2. API level: Since different platforms produce different results, tests must be executed on the Android O (API 26).
Testing:
The screenshot tests, when run on the specified Emulator, do the following sequentially:
1. Before running the test(s), app user data is cleared and device configuration is checked. This protects the cohesion of each test, making it possible to run them in any order, at a class suite level, individually or all at once. Any previously created comparison diff images[explained at step 5] will be deleted.
2. Device permissions are granted automatically before running each test, and hence there will be no need to run any gradle tasks for the same.
3. Launches the app and takes screenshots at the predetermined app states, and stored it in the device folder wikipedia-app-tests.
4. Compares them pixel-by-pixel to the reference screenshots which have been stored at androidTest/assets/espresso/screenshots.
5. The test tolerance has been set at 0%, which ensures that the test fails even at a difference of a pixel, between the reference and the subject screenshots. Should there be a failure, a comparison diff image is produced in the device folder wikipedia-app-tests/comparison
Example: Reference screenshot of the ‘History’ tab
Test screenshot, which has a different date format
The comparison diff produced:
Updating tests as needed:
When contributing code that alters UI, make sure you update the screenshot tests. Update should follow these steps:
1. Run the tests on the new screen changes. Your tests will fail as a result of the intentional UI change.
2. Make sure that the comparison image shows a diff only at the place where the intended change was made, and hasn’t introduced any unnecessary changes.
3. Use an online tool to crush the newly produced screenshots [found on your device folder wikipedia-app-tests]. Update the reference with this crushed image at androidTest/assets/espresso/screenshots.
Conventions[edit]
- Java code must run cleanly against Checkstyle
- JavaScript must run cleanly against JSHint
- We're working to be lint free! Currently Android Linting has many offenders but new contributions should help us progress towards zero.
- Python changes should run cleanly against flake8.
- All Android and JVM unit tests should pass.
- Git commit messages should conform to the Gerrit/Commit message guidelines
- Documentation should conform to the Wikipedia manual of style
- When making changes to any strings.xml, corresponding explanatory text should be added to values-qq/strings.xml
- Third-party libraries must meet acceptance criteria
- Patches should conform to our coding conventions
Help make it better![edit]
Pick a task from Phabricator:
- good first tasks / All tasks.
Testing:
See pending/recent code reviews:
Appendix[edit]
Tips[edit]
Optimizing Gradle builds[edit]
The Gradle build system requires a notoriously large amount of system resources, but there are ways of speeding it up and making sure it works consistently. Add the following lines to your ~/.gradle/gradle.properties file:
org.gradle.daemon=true org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.jvmargs=-Xmx2048M
WebView debugging in Chrome[edit]
Wikipedia for Android makes extensive use of WebViews. To debug WebView activity, navigate Google Chrome to chrome://inspect/#devices, then click on the topmost “inspect” link under “WebView in org.wikipedia.” From there you can debug the WebView like any other web site in Chrome.
Useful Gradle commands[edit]
If you prefer command line use the wrapper script in the root of the repo:
./gradlew
To run a clean debug build:
./gradlew -q clean assembleDevDebug
You can skip the clean part usually, which makes it much faster (from 1m:05s to 7s on my box):
./gradlew -q assembleDevDebug
To install build on device/emulator:
./gradlew -q installDevDebug
To see ProGuard output:
./gradlew clean --info proguardDevRelease
To run checkstyle:
./gradlew checkstyle
To run Lint:
./gradlew lintDevDebug
To refresh dependencies (usually not needed):
./gradlew --refresh-dependencies
To list dependencies:
./gradlew app:dependencies --configuration compile
Making apt work in Ubuntu 64-bit[edit]
If you're using 64-bit versions of Ubuntu, you might see this error when building with gradle: "Cannot run program '~/.android-sdk/build-tools/22.0.1/aapt': error=2, No such file or directory". To fix this error, install the following packages:
sudo apt-get install lib32stdc++6 lib32z1
Developer Settings[edit]
When using the dev build variant the developer settings are enabled automatically. For other flavors you could tap 7 times on the Wikipedia globe on the About page. Once enabled, you can open the developer settings by accessing the app settings, then tap on the icon on the right in the top toolbar.
Working with Vagrant and the Content Service[edit]
- To work with a local Vagrant instance, change the mediaWikiBaseUri developer setting and optionally disable mediaWikiBaseUriSupportsLangCode.
- To work with a local RESTBase instance, update RESTBaseUriFormat and optionally enable useRestbase_setManually.
- mediaWikiBaseUriSupportsLangCode controls whether the wiki language of mediaWikiBaseUri is prefixed.
- mediaWikiBaseUri and mediaWikiBaseUriSupportsLangCode are the second argument of RESTBaseUriFormat.
- useRestbase_setManually and useRestbase can affect whether a MediaWiki or RESTBase client is used internally. When a MediaWiki client is forced, RESTBaseUriFormat is unused.
- The Wiktionary domain is currently hardcoded. Ex: http://localhost:6927/en.wiktionary.org/v1/page/definition/dog
- It is recommend to fully terminate the app process after changing any of these settings.
Beta cluster config[edit]
Ex:
- https://en.wikipedia.beta.wmflabs.org/api/rest_v1/feed/featured/2016/06/01
- https://zh.wikipedia.beta.wmflabs.org/api/rest_v1/page/random/title
RESTBaseUriFormat (default): %1$s://%2$s/api/rest_v1/ mediaWikiBaseUriSupportsLangCode (default): enabled mediaWikiBaseUri (nondefault): https://wikipedia.beta.wmflabs.org
Local MediaWiki with production English Content Service config[edit]
Ex:
RESTBaseUriFormat (nondefault): %1$s://en.wikipedia.org/api/rest_v1/ mediaWikiBaseUriSupportsLangCode (nondefault): disabled mediaWikiBaseUri (nondefault): http://localhost:8080
Auth manager config[edit]
RESTBaseUriFormat (default): %1$s://%2$s/api/rest_v1/ mediaWikiBaseUriSupportsLangCode (nondefault): disabled mediaWikiBaseUri (nondefault): http://authmanager.wmflabs.org
Local Content Service config (broken)[edit]
Ex:
- http://localhost:6927/en.wikipedia.org/v1/page/most-read/2015/12/01
- http://192.168.1.10:6927/zh.wikipedia.org/v1/page/mobile-sections/%E4%B8%AD%E5%9C%8B
useRestbase_setManually (nondefault): enabled useRestbase (default): enabled RESTBaseUriFormat (nondefault): http://localhost:6927/%2$s/v1/ mediaWikiBaseUriSupportsLangCode (default): enabled mediaWikiBaseUri (default): https://wikipedia.org
This doesn't quite work at moment. The app attempts to use a RESTBase feed/ endpoint when page/ is needed for the Content Service.
Local Vagrant with local Content Service config (broken)[edit]
useRestbase_setManually (nondefault): enabled useRestbase (default): enabled RESTBaseUriFormat (nondefault): http://localhost:6927/%2$s/v1/ mediaWikiBaseUriSupportsLangCode (nondefault): disabled mediaWikiBaseUri (default): http://localhost:8080
This is currently broken for same reason the local Content Service config is.
See also[edit]
Scripts[edit]
Setup[edit]
cd scripts virtualenv -p python2 .env . .env/bin/activate pip install -r requirements.txt
Note: the lxml Python module seems to have system package dependencies on Debian and Ubuntu. If you're missing headers, try apt install libxml2-dev libxslt1-dev python-dev.
Troubleshooting Script Setup[edit]
On OS X a few of us ran into the issue that `libxml` couldn't get compiled. Running the following command helped:
xcode-select --install
If this still doesn't work, you might also want to try:
pip install sh jinja2 unicodecsv
Update bundled CSS files[edit]
The various CSS files for this project are generated by the *.less files found in the MobileFrontend and MobileApp MediaWiki extensions.
You'll need a MediaWiki installation, MediaWiki-Vagrant recommended, to generate them.
In the ./vagrant directory:
vagrant enable-role mobileapp mobilefrontend vagrant provision
Change the url in scripts/make-css-assets.bash to http://127.0.0.1:8080/w instead of the bits url, then run the script and test.
Note: your system may be unable to run the Android emulator and a Vagrant / VirtualBox instance simultaneously.
Update bundled JavaScript[edit]
Portions of JavaScript code run inside the WebView component that displays articles.
This code is prepackaged using Grunt, which must be re-run every time the master .js files are edited before building.
Preparing:
First, install the Grunt CLI tool:
npm install -g grunt-cli
Install dependencies for packaging:
cd www npm install
Building:
cd www grunt
This will produce output files under app/src/main/assets which will be included in the apk file.
Would like to try npm link with a local wikmedia-page-library repo? Please refer the [1] and then follow the steps above.
Update generated static data files[edit]
cd scripts python make-templates.py mv *.java ../app/src/main/java/org/wikipedia/staticdata/
Remote configuration[edit]
On startup, the app attempts to update its configuration by checking a JSON config file updated as part of the MobileApp extension. This remote configuration file is typically used for things like managing EventLogging sampling rates or rolling out new features incrementally.
On Wikimedia production wikis, the file is located at https://<host>/static/current/extensions/MobileApp/config/android.json.
On the Beta Cluster or a local MediaWiki environment (such as MediaWiki-Vagrant) with the MobileApp extension, the file is available at //<host>/w/extensions/MobileApp/config/android.json.
See T162164 for details on the treatment of static files in the Wikimedia production environment.
Purging the cached configuration (note: requires production shell access)[edit]
After a configuration update is deployed, making it available in production before the old version naturally expires out of cache will require a purge.
To do so:
ssh tin echo "https://www.wikimedia.org/static/current/extensions/MobileApp/config/android.json" | mwscript purgeList.php --wiki=aawiki
String resource translations[edit]
The app's string resources are translated by the volunteer community at Translatewiki.net. See Translation of app string resources (and in particular the section TWN sync for Android) for details.
Alpha build server[edit]
See Alpha build server.