User:DBrant (WMF)/Goodies

Uncle Dmitry's bag of goodies. You're welcome.

Apps team pages

 * Mobile team portal
 * Apps team page
 * Release process
 * Why apps?
 * Release history (update with every release)
 * Roles, responsibilities
 * Development cycle
 * Coding conventions
 * All subpages under mobile apps
 * Design guidelines
 * App readme and FAQ and Play Store description
 * Apps/Desktop feature matrix

Plans and goals

 * Offline Library
 * Community wishlist item.
 * Synchronized reading lists
 * Short descriptions (original RfC on Wikidata)
 * New editor experiences.
 * Running on Chromebooks
 * Feeds and Navigation.
 * The iOS team's work on Notifications and enhanced Nearby
 * Planning for: 2016-Q1 | 2015-Q4
 * Goals for: 2016-Q1 | 2015-Q4

Phabricator boards

 * Android app
 * iOS app development and backlog.
 * Content service
 * Kanban thereof
 * Roadmap board
 * Reading admin board
 * Project creators
 * Guidelines for new projects
 * Calendar board (for marking vacation)

Deprecated

 * Mobile app goals
 * General/unknown app
 * Older mobile apps board

Android
$ brew install jd-gui $ brew install dex2jar $ d2j-dex2jar  -o  $ adb tcpip 5555 $ adb connect :5555 $ adb -s :5555 usb $ adb shell $ netcfg $ adb shell $ am broadcast -a android.intent.action.BOOT_COMPLETED The above may actually cause your device to reboot, in which case you'll get the event when the device finished booting! ;) $ keytool.exe -list -printcert -jarfile [filename].apk
 * Material Design icons
 * Animated vector drawable designer
 * Screen sharing: Android Screen Monitor (ASM)
 * Screen recording for Mac
 * Memory profiling: Eclipse MAT
 * Inspect the view hierarchy of any running app: uiautomatorviewer (ANDROID_SDK/tools/bin/uiautomatorviewer)
 * Decompile: APKtool, dex2jar
 * Convert SVG to vector drawables
 * Activity lifecycle
 * Porter/Duff modes
 * ImageView scaleTypes
 * Annotations
 * Device metrics
 * Install Google apps in Genymotion.
 * To do ADB over the network:
 * To switch back to ADB over USB:
 * To get the IP address of the device:
 * To simulate a BOOT_COMPLETED event:
 * To examine the fingerprints of the signing key that was used to sign an APK (the keytool executable is found in the JDK folder):

Testing the INSTALL_REFERRER event
https://play.google.com/store/apps/details?id=org.wikipedia&referrer=utm_source%3Dtest_source%26utm_medium%3Dtest_medium%26utm_term%3Dtest-term%26utm_content%3Dtest_content%26utm_campaign%3Dtest_name
 * Set the build variant to Production (prodDebug) or Beta (betaDebug).
 * Make sure the app is uninstalled on your device.
 * On your device, click the link that contains your desired Install Referrer parameters, which should lead to the Play Store:

(In the above link, make sure to specify the Production or Beta package name, corresponding to the variant you're testing.)


 * When the Play Store opens, do not install the app from the Play Store, but instead just launch or debug the app from Android Studio.
 * The Install Referrer event should be received!

MySQL

 * Linux/Windows/Mac: MySQL Workbench
 * (Windows only) Establish an ssh tunnel to stat1006:  (note the port number "222", so that it doesn't conflict with any local SSH server that might be running)
 * SSH Hostname: localhost, SSH Username: dbrant, MySQL Hostname:, Username: research
 * Get the password for the "research" user from.
 * Note, when installing on Windows, and you find that Workbench fails to connect to Wikimedia's stat1006 over the Bastion tunnel, try using this fork of paramiko.
 * Also, to make text fields appear properly, go to Edit -> Preferences, SQL Editor -> SQL Execution, and check the box "Treat BINARY/VARBINARY as nonbinary character string."
 * Mac only: Sequel Pro
 * MySQL Host:, Username: research, SSH Host:
 * Get the password for the "research" user from.

Gerrit / Github repos

 * Make sure to upload your public key to Gerrit.
 * Make sure to install git review.
 * all open patches in Android app
 * all open patches in Content service
 * all open patches in MobileApp extension
 * all open patches in java-mwapi
 * all patches by me
 * Gerrit reviewer bot
 * Gerrit watched projects
 * Wikimedia-page-library

Git Review workflow
When modifying/amending an existing patch:

...make changes...

Git
$ git remote add upstream https://github.com/foo/bar.git $ git fetch upstream $ git checkout master $ git merge upstream/master
 * Bash git prompt
 * Git completion
 * To sync a fork with the upstream repo:

API

 * Documentation
 * Api Sandbox (enwiki)
 * Api Sandbox (wikidata)
 * MediaWiki REST endpoints.
 * Parsoid DOM spec
 * CirrusSearch

Services
https://en.wikipedia.org/api/rest_v1/page/title/Cat https://en.wikipedia.org/api/rest_v1/page/summary/Cat https://en.wikipedia.org/api/rest_v1/page/html/Cat https://en.wikipedia.org/api/rest_v1/page/random/title https://en.wikipedia.org/api/rest_v1/page/random/summary https://en.wikipedia.org/api/rest_v1/page/mobile-html/Cat Feed: https://en.wikipedia.org/api/rest_v1/feed/featured/2017/04/01 https://en.wikipedia.org/api/rest_v1/feed/announcements https://en.wikipedia.org/api/rest_v1/feed/trending/edits https://en.wikipedia.org/api/rest_v1/feed/onthisday/all/08/07 https://en.wikipedia.org/api/rest_v1/feed/onthisday/births/08/07 Deprecated: https://en.wikipedia.org/api/rest_v1/page/mobile-sections/Cat https://en.wikipedia.org/api/rest_v1/page/mobile-sections-lead/Cat https://en.wikipedia.org/api/rest_v1/page/mobile-sections-remaining/Cat Testing: http://apps.wmflabs.org/en.wikipedia.org/v1/page/summary/Cat http://apps.wmflabs.org/en.wikipedia.org/v1/page/mobile-html/Cat
 * RESTBase for apps
 * Documentation
 * Content service in gerrit
 * Reading List service test instance.
 * URL shortener
 * Examples:

Running locally
http://192.168.1.218:6927/en.wikipedia.org/v1/page/summary/Cat

http://localhost:8889/en.wikipedia.org/v1/feed/announcements

CI / Jenkins / Zuul

 * integration/config repository (on GitHub).
 * Master Zuul configuration. Controls reviewer (+1 / +2) access to projects, etc.
 * Jenkins configuration for mobile projects. Sets up CI jobs/sequences for our projects.


 * Main Jenkins dashboard.
 * Search of the dashboard for everything about "apps"
 * Add a comment of "recheck" to force Jenkins to reevaluate a patch.

Remember to always !log in #wikimedia-qa when performing manual commands via Jenkins control panel.
 * Zuul status.
 * Known issues
 * Gearman deadlock

Wikitech / OpenStack / Cloud services

 * After creating an account, make sure to upload your public key.
 * Deployment-prep project, where new tools and experimental features are hosted before going to production. (Hosts testing eventlogging server, among other things.)
 * Mobile project. (Hosts a few instances that we use for testing certain features.)
 * Recommendations API project. (Some further relevant instances for our features.)
 * List of all projects.
 * Nova Resource category.
 * If you are a member or administrator of a project, you can use Horizon to monitor and administer it. (Requires your Wikitech account to have 2FA set up.)

Toolforge

 * Admin console
 * Portal / quickstart page
 * Log in with, then enter

MEP (Modern event platform)
Debugging URLs:

https://stream-beta.wmflabs.org/v2/ui/#/?streams=android.user_contribution_screen

https://stream-beta.wmflabs.org/v2/ui/#/?streams=eventgate-analytics-external.error.validation

Legacy
$ ssh deployment-eventlog05.eqiad.wmflabs $ tail -f /srv/log/eventlogging/all-events.log | grep "MobileWikiApp"
 * Wiki page
 * Administration
 * Data access guidelines
 * Event logging:  (formerly stat1003), Hadoop/Hive:   (formerly stat1005, formerly stat1002)
 * For access to stat1006, you need to be part of the "researchers" group, and for stat1007, you need to be in the "analytics-privatedata-users" group.
 * Event logging schema list
 * Event logging help for production and beta.
 * Event logging testing on labs:
 * When creating a new schema, make sure to populate the Talk page with the SchemaDoc template (copy from an existing schema).
 * For whitelisting specific fields within the schema, make a Gerrit patch to the puppet repository, in this file. We generally want to whitelist all fields apart from the standard eventlogging capsule.
 * relevant wiki page
 * Hive help
 * Hive FAQ and samples
 * Hue
 * Refinery source code for daily/monthly uniques

Analytics

 * SWAP: Python notebooks integrated into eventlogging.

Editing testing
This is to test any exceptional conditions when submitting edits.

Triggering AbuseFilter

 * Open any article on Test Wiki.
 * Try to submit any edit that contains the text "the abuse filter will block this".

Triggering Spam filter

 * Open any article on English Wikipedia (preferably your user sandbox page).
 * Try to submit any edit that contains the text "https://example.xyz".

Releases server (formerly releases1002, formerly releases1001, formerly Bromine, formerly Caesium)
To copy stuff over: $ scp [file] releases.discovery.wmnet:~/[file] $ scp app.apk releases.discovery.wmnet:/srv/org/wikimedia/releases/mobile/android/wikipedia/alphas/app.apk
 * https://releases.wikimedia.org/mobile/android/wikipedia/
 * releases.discovery.wmnet (alias for releases2002.codfw.wmnet)

Vagrant
Once installed and launched, here are some useful links:
 * http://en.wiki.local.wmftest.net:8080/wiki/Main_Page
 * http://en.wiki.local.wmftest.net:8080/wiki/Special:ApiSandbox
 * http://fr.wiki.local.wmftest.net:8080/wiki/Accueil
 * http://wikidata.wiki.local.wmftest.net:8080/wiki/Special:SpecialPages
 * http://wikidata.wiki.local.wmftest.net:8080/wiki/Q1
 * http://wikidata.wiki.local.wmftest.net:8080/wiki/Property:P1

people.wikimedia.org

 * This is hosted on  (previously  )
 * https://people.wikimedia.org/~dbrant/
 * Help on Wikitech

TranslateWiki

 * Translation for app strings
 * Direct link to a specific string in a specific language: https://translatewiki.net/wiki/Wikimedia:Wikipedia-android-strings-notification_syncing_title/nb

Wikidata

 * autodesc
 * WDQ
 * Documentation
 * WikiShootMe - find nearby articles that need a picture

Bintray / Maven / Jcenter

 * My maven entry for Zimdroid.
 * My maven entry for wikimedia-android-data-client.

.ssh config
Host bastcloud HostName bastion.wmcloud.org User dbrant IdentityFile ~/.ssh/id_rsa Host *.wmcloud.org *.wikimedia.cloud !bastion.wmcloud.org !primary.bastion.wmcloud.org User dbrant IdentityFile ~/.ssh/id_rsa ProxyJump bastcloud Host bastlabs HostName primary.bastion.wmflabs.org User dbrant IdentityFile ~/.ssh/id_rsa Host *.wmflabs *.wmflabs.org !primary.bastion.wmflabs.org User dbrant IdentityFile ~/.ssh/id_rsa ProxyJump bastlabs Host bastprod HostName bast1003.wikimedia.org User dbrant IdentityFile ~/.ssh/id_rsa_prod Host *.wmnet *.wikimedia.org !bast*.wikimedia.org !gerrit.wikimedia.org !gitlab.wikimedia.org User dbrant ProxyJump bastprod IdentityFile ~/.ssh/id_rsa_prod Host bast*.wikimedia.org User dbrant IdentityFile ~/.ssh/id_rsa_prod Host gerrit.wikimedia.org gitlab.wikimedia.org User dbrant IdentityFile ~/.ssh/id_rsa_gerrit
 * 1) Bastion host for wmfcloud
 * 1) Bastion host for wmflabs
 * 1) Production bastion host.
 * 1) Proxy all connections to internal servers through bastion host.
 * 1) Connection to actual bastion host.

The above file is probably a bit more complex than it needs to be, but it has the following points of interest:
 * Uses a different key file for production vs wmflabs.
 * Uses a still different key for Gerrit and Gitlab.

On macOS 10.12 and above, you may need to add the following (since it no longer adds the keys to ssh-agent by default):

Host * UseKeychain yes AddKeysToAgent yes

Onboarding/Fundamentals

 * Gerrit
 * MediaWiki-Vagrant
 * Dotfiles
 * Mobile Web dev sessions
 * Research FAQ

Administrative

 * Production shell access
 * NOC
 * Labs mobile instances
 * Help
 * Deployments
 * Test Wiki
 * Block, Unblock
 * Mobile report card (limn)
 * Search metrics
 * Wikipedia manual of style
 * Wikimedia blog
 * JetBrains licenses

Design

 * Wikimedia design style guide.
 * Brand materials.
 * Wikipedia and Wikimedia visual identity guidelines, and list of trademarks.

Disabling startup processes on macOS
Background processes that are started at boot time are in the following directories:

/Library/LaunchDaemons /Library/LaunchAgents

The above directories contain .plist files that describe the process(es) to be launched. It's usually possible to just delete (with sudo) any launch file that's no longer needed. However, certain files might be protected by SIP, and not removable even with sudo. In this case, follow these steps:


 * Boot into recovery mode (for an M1 MacBook, hold and keep holding the power button), and run  in a console.
 * Reboot again into recovery mode.
 * Go to the Disk Utility and mount the Data volume.
 * Open a console and delete the desired files. The launch .plist files will now be located in these directories:

/Volumes/Macintosh HD/Library/LaunchDaemons /Volumes/Macintosh HD/Library/LaunchAgents /Volumes/Data/Library/LaunchDaemons /Volumes/Data/Library/LaunchAgents

(you may need to remount the partition as writable: )
 * Re-enable SIP:
 * Reboot normally.

Miscellaneous

 * My gists
 * Typographical symbols
 * Emoji cheat sheet
 * Manage a website using git.
 * FIT file tools

curl -d "param=value&param2=value2" -X POST "https://..." mogrify -format jpg -quality 90 *.png mogrify -format jpg -quality 90 -thumbnail 512 *.png exiftool -AllDates-= -AllDates="2008:01:01 12:00:00" -overwrite_original -r. exiftool -gps:all= -xmp:geotag= image.jpg -overwrite_original * exiftool *.jpg -Copyright="Copyright (c) Dmitry Brant." ffmpeg -r 60 -f image2 -s 600x360 -start_number 0 -i frame%d.png -vcodec libx264 -crf 18 -pix_fmt yuv420p test.mp4 (-r = frame rate, -crf = quality (lower is better))
 * Send a POST request using curl:
 * Convert a batch of PNG images to JPG:
 * Convert a batch of PNG images to JPG, and resize to thumbnail:
 * Encode an Exif date tag into photos (recursively, in a whole directory):
 * Remove the GPS Exif tag from photos:
 * Add Copyright tag to all JPG images in directory:
 * Create video (x264) from a series of still frames (png):

ffmpeg -r 60 -f image2 -s 1280x720 -start_number 0 -i frame%d.png -vcodec libtheora -qscale:v 7 test.ogv python CrossMap.py bam hg19ToHg38.over.chain WGC074699D_combined.chr17.bam chr17.bam ./otf2bdf -p 10 -r 96 -o font.bdf font.ttf (-p = font size, -r = screen dpi) qemu-img convert -f raw -O qcow2 image.img image.qcow2 qemu-img convert -f vmdk -O raw image.vmdk image.img python .\sqlmap.py -u "https://url/?param1=value1&param2=value2" -p "param1,param2" python .\sqlmap.py -u "https://url/" --data "param1=value1" -p "param1" --method POST
 * Convert (cross map) a genomic file (BAM) from one reference genome to another:
 * Create a bitmap font for use with Python PIL from a TrueType font:
 * Download the otf2bdf tool: http://sofia.nmsu.edu/~mleisher/Software/otf2bdf/
 * Use the pilfont utility, or do it directly in Python.
 * Convert an emulator disk image from one format to another (using QEMU)
 * Use sqlmap for basic pentesting:

zsteg -a filename.png zsteg -E [name] filename.png
 * Steganography for PNG files: zsteg


 * File format archive
 * Online graphing calculator
 * Online hex editor
 * List of unofficial Wikipedia apps

Teachable moments

 * Community wishlist 2019
 * Community wishlist 2017
 * Community wishlist 2016
 * Community wishlist 2015
 * Gather
 * RfC to disable Gather
 * Our proposal to the RfC (see talk page)
 * Moderation criteria
 * Extension talk page
 * Old RfC for user-specific lists
 * Selfiepocalypse
 * why photo uploads from mobile were disabled.
 * proposal to shut it down
 * village pump discussion
 * Risker's checklist for new content-creation extensions
 * Antandrus's observations on Wikipedia behavior
 * Sue Gardner's guide to working with online communities
 * Sue Gardner's patterns for change that work
 * JMinor's essays on product management.
 * Refactoring is the enemy of the product owner.
 * Do the least amount of work while still delivering value to the user (or at least strike a balance between user-facing goals and technical debt).
 * Don't let perfect be the enemy of good.