User:DBrant (WMF)/Goodies

From mediawiki.org

Uncle Dmitry's bag of goodies. You're welcome.

Apps team pages[edit]

Plans and goals[edit]

Phabricator boards[edit]

Deprecated[edit]

Android[edit]

$ brew install jd-gui
$ brew install dex2jar
$ d2j-dex2jar <input.dex> -o <output.jar>

$ adb tcpip 5555
$ adb connect <ip-address>:5555
  • To switch back to ADB over USB:
$ adb -s <ip-address>:5555 usb
  • To get the IP address of the device:
$ adb shell
$ netcfg
  • To simulate a BOOT_COMPLETED event:
$ adb shell
$ am broadcast -a android.intent.action.BOOT_COMPLETED

The above may actually cause your device to reboot, in which case you'll get the event when the device finished booting! ;)

  • To examine the fingerprints of the signing key that was used to sign an APK (the keytool executable is found in the JDK folder):
$ keytool.exe -list -printcert -jarfile [filename].apk

Testing the INSTALL_REFERRER event[edit]

  • Set the build variant to Production (prodDebug) or Beta (betaDebug).
  • Make sure the app is uninstalled on your device.
  • On your device, click the link that contains your desired Install Referrer parameters, which should lead to the Play Store:

https://play.google.com/store/apps/details?id=org.wikipedia&referrer=utm_source%3Dtest_source%26utm_medium%3Dtest_medium%26utm_term%3Dtest-term%26utm_content%3Dtest_content%26utm_campaign%3Dtest_name

(In the above link, make sure to specify the Production or Beta package name, corresponding to the variant you're testing.)

  • When the Play Store opens, do not install the app from the Play Store, but instead just launch or debug the app from Android Studio.
  • The Install Referrer event should be received!

MySQL[edit]

  • Linux/Windows/Mac: MySQL Workbench
    • (Windows only) Establish an ssh tunnel to stat1006: $ ssh -L 222:localhost:22 stat1006.eqiad.wmnet (note the port number "222", so that it doesn't conflict with any local SSH server that might be running)
    • SSH Hostname: localhost, SSH Username: dbrant, MySQL Hostname: analytics-slave.eqiad.wmnet, Username: research
    • Get the password for the "research" user from stat1006/etc/mysql/conf.d/research-client.cnf.
    • Note, when installing on Windows, and you find that Workbench fails to connect to Wikimedia's stat1006 over the Bastion tunnel, try using this fork of paramiko.
    • Also, to make text fields appear properly, go to Edit -> Preferences, SQL Editor -> SQL Execution, and check the box "Treat BINARY/VARBINARY as nonbinary character string."
  • Mac only: Sequel Pro
    • MySQL Host: analytics-slave.eqiad.wmnet, Username: research, SSH Host: stat1006.eqiad.wmnet
    • Get the password for the "research" user from stat1006/etc/mysql/conf.d/research-client.cnf.

Gerrit / Github repos[edit]

Git Review workflow[edit]

When modifying/amending an existing patch:

git review -d [changenumber]

...make changes...

git commit --all --amend

git review -R

Git[edit]

$ git remote add upstream https://github.com/foo/bar.git
$ git fetch upstream
$ git checkout master
$ git merge upstream/master

API[edit]

Services[edit]

https://en.wikipedia.org/api/rest_v1/page/title/Cat
https://en.wikipedia.org/api/rest_v1/page/summary/Cat
https://en.wikipedia.org/api/rest_v1/page/html/Cat
https://en.wikipedia.org/api/rest_v1/page/random/title
https://en.wikipedia.org/api/rest_v1/page/random/summary
https://en.wikipedia.org/api/rest_v1/page/mobile-html/Cat

Feed: 

https://en.wikipedia.org/api/rest_v1/feed/featured/2017/04/01
https://en.wikipedia.org/api/rest_v1/feed/announcements
https://en.wikipedia.org/api/rest_v1/feed/trending/edits
https://en.wikipedia.org/api/rest_v1/feed/onthisday/all/08/07
https://en.wikipedia.org/api/rest_v1/feed/onthisday/births/08/07

Deprecated: 

https://en.wikipedia.org/api/rest_v1/page/mobile-sections/Cat
https://en.wikipedia.org/api/rest_v1/page/mobile-sections-lead/Cat
https://en.wikipedia.org/api/rest_v1/page/mobile-sections-remaining/Cat

Testing: 

http://apps.wmflabs.org/en.wikipedia.org/v1/page/summary/Cat
http://apps.wmflabs.org/en.wikipedia.org/v1/page/mobile-html/Cat

Running locally[edit]

http://192.168.1.218:6927/en.wikipedia.org/v1/page/summary/Cat

http://localhost:8889/en.wikipedia.org/v1/feed/announcements

CI / Jenkins / Zuul[edit]

  • Main Jenkins dashboard.
  • Search of the dashboard for everything about "apps"
  • Add a comment of "recheck" to force Jenkins to reevaluate a patch.

Remember to always !log in #wikimedia-qa when performing manual commands via Jenkins control panel.

Wikitech / OpenStack / Cloud services[edit]

  • After creating an account, make sure to upload your public key.
  • Deployment-prep project, where new tools and experimental features are hosted before going to production. (Hosts testing eventlogging server, among other things.)
  • Mobile project. (Hosts a few instances that we use for testing certain features.)
  • Recommendations API project. (Some further relevant instances for our features.)
  • List of all projects.
  • Nova Resource category.
  • If you are a member or administrator of a project, you can use Horizon to monitor and administer it. (Requires your Wikitech account to have 2FA set up.)

Toolforge[edit]

Event logging[edit]

MEP (Modern event platform)[edit]

Debugging URLs:

https://stream-beta.wmflabs.org/v2/ui/#/?streams=android.user_contribution_screen

https://stream-beta.wmflabs.org/v2/ui/#/?streams=eventgate-analytics-external.error.validation

Legacy[edit]

$ ssh deployment-eventlog05.eqiad.wmflabs
$ tail -f /srv/log/eventlogging/all-events.log | grep "MobileWikiApp"

Analytics[edit]

  • SWAP: Python notebooks integrated into eventlogging.

Editing testing[edit]

This is to test any exceptional conditions when submitting edits.

Triggering AbuseFilter[edit]

  • Open any article on Test Wiki.
  • Try to submit any edit that contains the text "the abuse filter will block this".

Triggering Spam filter[edit]

  • Open any article on English Wikipedia (preferably your user sandbox page).
  • Try to submit any edit that contains the text "https://example.xyz".

Releases server (formerly releases1002, formerly releases1001, formerly Bromine, formerly Caesium)[edit]

To copy stuff over: 

$ scp [file] releases.discovery.wmnet:~/[file]
$ scp app.apk releases.discovery.wmnet:/srv/org/wikimedia/releases/mobile/android/wikipedia/alphas/app.apk

Vagrant[edit]

Once installed and launched, here are some useful links:

people.wikimedia.org[edit]

TranslateWiki[edit]

Wikidata[edit]

Bintray / Maven / Jcenter[edit]

.ssh config[edit]

# Bastion host for wmfcloud
Host bastcloud
    HostName bastion.wmcloud.org
    User dbrant
    IdentityFile ~/.ssh/id_rsa

Host *.wmcloud.org *.wikimedia.cloud !bastion.wmcloud.org !primary.bastion.wmcloud.org
    User dbrant
    IdentityFile ~/.ssh/id_rsa
    ProxyJump bastcloud

# Bastion host for wmflabs
Host bastlabs
    HostName primary.bastion.wmflabs.org
    User dbrant
    IdentityFile ~/.ssh/id_rsa

Host *.wmflabs *.wmflabs.org !primary.bastion.wmflabs.org
    User dbrant
    IdentityFile ~/.ssh/id_rsa
    ProxyJump bastlabs

# Production bastion host.
Host bastprod
    HostName bast1003.wikimedia.org
    User dbrant
    IdentityFile ~/.ssh/id_rsa_prod

# Proxy all connections to internal servers through bastion host.
Host *.wmnet *.wikimedia.org !bast*.wikimedia.org !gerrit.wikimedia.org !gitlab.wikimedia.org
    User dbrant
    ProxyJump bastprod
    IdentityFile ~/.ssh/id_rsa_prod

# Connection to actual bastion host.
Host bast*.wikimedia.org
    User dbrant
    IdentityFile ~/.ssh/id_rsa_prod

Host gerrit.wikimedia.org gitlab.wikimedia.org
    User dbrant
    IdentityFile ~/.ssh/id_rsa_gerrit

The above file is probably a bit more complex than it needs to be, but it has the following points of interest:

  • Uses a different key file for production vs wmflabs.
  • Uses a still different key for Gerrit and Gitlab.

On macOS 10.12 and above, you may need to add the following (since it no longer adds the keys to ssh-agent by default): 

Host *
UseKeychain yes
AddKeysToAgent yes

Onboarding/Fundamentals[edit]

Administrative[edit]

Design[edit]

Administrative tips[edit]

Disabling startup processes on macOS[edit]

Background processes that are started at boot time are in the following directories: 

/Library/LaunchDaemons
/Library/LaunchAgents

The above directories contain .plist files that describe the process(es) to be launched. It's usually possible to just delete (with sudo) any launch file that's no longer needed. However, certain files might be protected by SIP, and not removable even with sudo. In this case, follow these steps:

  • Boot into recovery mode (for an M1 MacBook, hold and keep holding the power button), and run csrutil disable in a console.
  • Reboot again into recovery mode.
  • Go to the Disk Utility and mount the Data volume.
  • Open a console and delete the desired files. The launch .plist files will now be located in these directories:
/Volumes/Macintosh HD/Library/LaunchDaemons
/Volumes/Macintosh HD/Library/LaunchAgents
/Volumes/Data/Library/LaunchDaemons
/Volumes/Data/Library/LaunchAgents

(you may need to remount the partition as writable: mount -uw /)

  • Re-enable SIP: csrutil enable
  • Reboot normally.

Miscellaneous[edit]

  • Send a POST request using curl:
curl -d "param=value&param2=value2" -X POST "https://..."
  • Convert a batch of PNG images to JPG:
 mogrify -format jpg -quality 90 *.png
  • Convert a batch of PNG images to JPG, and resize to thumbnail:
 mogrify -format jpg -quality 90 -thumbnail 512 *.png
  • Encode an Exif date tag into photos (recursively, in a whole directory):
 exiftool -AllDates-= -AllDates="2008:01:01 12:00:00" -overwrite_original -r .
  • Remove the GPS Exif tag from photos:
 exiftool -gps:all= -xmp:geotag= image.jpg -overwrite_original *
  • Add Copyright tag to all JPG images in directory:
 exiftool *.jpg -Copyright="Copyright (c) Dmitry Brant."
  • Create video (x264) from a series of still frames (png):
 ffmpeg -r 60 -f image2 -s 600x360 -start_number 0 -i frame%d.png -vcodec libx264 -crf 18 -pix_fmt yuv420p test.mp4
 (-r = frame rate, -crf = quality (lower is better))

 ffmpeg -r 60 -f image2 -s 1280x720 -start_number 0 -i frame%d.png -vcodec libtheora -qscale:v 7 test.ogv
  • Convert (cross map) a genomic file (BAM) from one reference genome to another:
 python CrossMap.py bam hg19ToHg38.over.chain WGC074699D_combined.chr17.bam chr17.bam

 ./otf2bdf -p 10 -r 96 -o font.bdf font.ttf
 (-p = font size, -r = screen dpi)
  • Convert an emulator disk image from one format to another (using QEMU)
 qemu-img convert -f raw -O qcow2 image.img image.qcow2
 qemu-img convert -f vmdk -O raw image.vmdk image.img
  • Use sqlmap for basic pentesting:
python .\sqlmap.py -u "https://url/?param1=value1&param2=value2" -p "param1,param2"
python .\sqlmap.py -u "https://url/" --data "param1=value1" -p "param1" --method POST
  • Steganography for PNG files: zsteg
zsteg -a filename.png
zsteg -E [name] filename.png

Teachable moments[edit]

Retrieved from "https://www.mediawiki.org/w/index.php?title=User:DBrant_(WMF)/Goodies&oldid=6078728"