User:JHernandez (WMF)/Test Html to Wikitext

Converted from https://gitlab.com/aklapper/phabricator-lorem-ipsum/-/raw/master/how-to-phab-tutorial-videos.html

List of steps performed in 2020 to set up a local Phabricator installation on a local Fedora machine and make it look close enough to the production instance and to make it have some data, with the goal to create Wikimedia Phabricator tutorial videos. See https://phabricator.wikimedia.org/T214522 for more context.


 * First follow the installation guide which basically means:
 * sudo dnf install httpd mariadb-server php php-gd php-mbstring php-mysqlnd php-opcache php-pecl-apcu-bc php-pecl-zip php-xml
 * sudo dnf install php-process (needed for running the daemons which say "posix PHP extension not installed")
 * sudo systemctl restart httpd.service
 * sudo systemctl restart mariadb.service
 * Open browser and go to http://localhost/ to check if httpd is up
 * sudo mkdir /var/www/html/phab/
 * cd /var/www/html/phab/
 * sudo git clone https://github.com/phacility/libphutil.git
 * sudo git clone https://github.com/phacility/arcanist.git
 * sudo git clone https://github.com/phacility/phabricator.git
 * Now follow the configuration guide:
 * On this machine, my user account is called 'ak'.
 * sudo vim /etc/httpd/conf/httpd.conf :    ServerName phab.localhost   DocumentRoot /var/www/html/phab/phabricator/webroot      Options Indexes FollowSymLinks     AllowOverride All     Require all granted    RewriteEngine on   RewriteRule ^/rsrc/(.*)    -                       [L,QSA]   RewriteRule ^/favicon.ico  -                       [L,QSA]    RewriteRule ^/(.*)$       /index.php?__path__=/$1  [B,L,QSA]   ErrorLog /var/log/httpd/error_log 
 * sudo mysql
 * mysql> CREATE USER 'ak'@'localhost.foo' IDENTIFIED BY '' ;
 * mysql> GRANT ALL PRIVILEGES ON *.* TO 'ak'@'localhost.foo' IDENTIFIED BY '' ;

Maria / Maria / maria@example.com, Kate / Kate / kate@example.com , Yves / Yves / yves@example.com , Cassidy / Cassidy / cassidy@example.com , Finley / Finley / finley@example.com , and for the sake of faking it also as a standard user gerritbot / Gerrit / gerritbot@example.com. (See Configuring accounts how to log into these accounts from a console, basically:  /var/www/html/phab/phabricator/bin/auth recover someUsername and then use the shown link.) Log in as Gerritbot, go to its profile, and set the user profile image to https://phab.wmfusercontent.org/file/data/ctemipegegt3vq2bjkqs/PHID-FILE-x4lhwocmrnntw7ph3wwp/profile. START TRANSACTION; use phabricator_project; TRUNCATE TABLE edge; TRUNCATE TABLE edgedata; TRUNCATE TABLE project; TRUNCATE TABLE project_column; TRUNCATE TABLE project_columnposition; TRUNCATE TABLE project_columntransaction; TRUNCATE TABLE project_customfieldnumericindex; TRUNCATE TABLE project_customfieldstorage; TRUNCATE TABLE project_customfieldstringindex; TRUNCATE TABLE project_datasourcetoken; TRUNCATE TABLE project_project_fdocument; TRUNCATE TABLE project_project_ffield; TRUNCATE TABLE project_project_fngrams; TRUNCATE TABLE project_project_fngrams_common; TRUNCATE TABLE project_slug; TRUNCATE TABLE project_transaction; TRUNCATE TABLE project_trigger; TRUNCATE TABLE project_triggertransaction; TRUNCATE TABLE project_triggerusage; use phabricator_maniphest; TRUNCATE TABLE edge; TRUNCATE TABLE edgedata; TRUNCATE TABLE maniphest_customfieldnumericindex; TRUNCATE TABLE maniphest_customfieldstorage; TRUNCATE TABLE maniphest_customfieldstringindex; TRUNCATE TABLE maniphest_nameindex; TRUNCATE TABLE maniphest_task; TRUNCATE TABLE maniphest_task_fdocument; TRUNCATE TABLE maniphest_task_ffield; TRUNCATE TABLE maniphest_task_fngrams; TRUNCATE TABLE maniphest_task_fngrams_common; TRUNCATE TABLE maniphest_transaction; TRUNCATE TABLE maniphest_transaction_comment; use phabricator_user; TRUNCATE TABLE edge; TRUNCATE TABLE edgedata; TRUNCATE TABLE phabricator_session; COMMIT; gsettings set org.gnome.settings-daemon.plugins.media-keys max-screencast-length "uint32 600" for file in videoXX-YY-raw.webm; do ffmpeg -i "$file" -b:v 2M -filter:v "crop=1675:980:0:100" "${file/%-raw.webm/-cropped.webm}"; done for file in *.svg; do convert ${file} ${file}.png; done ffmpeg -loop 1 -i slide.png -c:v libvpx-vp9 -b:v 2M -t 7 -strict experimental -shortest videoXX-YY-cropped.webm for file in video*.w*; do echo "$file"; ffmpeg -i "$file" 2>&1 | grep Duration; done for file in video*-cropped.wav; do ffmpeg -i "${file/%.wav/-cropped.webm}" -i "$file" -c:v copy -c:a opus -strict experimental "${file/%.wav/-av.webm}"; done ffmpeg -y -i videoXX-rc.webm -c:v libvpx-vp9 -pass 1 -b:v 1000K -threads 1 -speed 4 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -an -f webm /dev/null ffmpeg -i videoXX-rc.webm -c:v libvpx-vp9 -pass 2 -b:v 1000K -threads 1 -speed 0 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -c:a libopus -b:a 64k -ac 1 -f webm videoXX-final-full.webm ffmpeg -y -i videoXX-rc.webm -c:v libvpx-vp9 -pass 1 -b:v 500K -threads 1 -speed 4 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -an -f webm /dev/null ffmpeg -i videoXX-rc.webm -filter:v scale=1060:-1 -c:v libvpx-vp9 -pass 2 -b:v 500K -threads 1 -speed 0 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -c:a libopus -b:a 64k -ac 1 -f webm videoXX-final-reduced.webm
 * sudo /var/www/html/phab/phabricator/bin/config set mysql.host "localhost"
 * sudo /var/www/html/phab/phabricator/bin/config set mysql.user ak
 * sudo /var/www/html/phab/phabricator/bin/config set mysql.pass ""
 * sudo systemctl restart mariadb.service
 * sudo /var/www/html/phab/phabricator/bin/storage upgrade --user "ak" --password "" --force
 * sudo /var/www/html/phab/phabricator/bin/config set phabricator.base-uri " http://phab.localhost "
 * sudo systemctl restart mariadb.service
 * sudo systemctl restart httpd.service
 * In /etc/php.ini change post_max_size = 8M to post_max_size = 32M
 * Go to http://phab.localhost/ and log in as admin
 * Implement WMF customizations:
 * Copy local.json to the location /var/www/html/phab/phabricator/conf/local/local.json which adds WMF Priority and status values, custom wordmark, enables serious business mode, same color scheme, etc by
 * Download custom WMF Phab logo and upload it via http://phab.localhost/file/upload, then open that file in browser to get its actual URL, then copy its internal name into /var/www/html/phab/phabricator/conf/local/local.json for the ui.logo value
 * Set http://phab.localhost/config/edit/ui.header-color/ to Blue as I'm too stupid to find out how to do that in local.json
 * Set http://phab.localhost/config/edit/maniphest.custom-field-definitions/ to the same as production
 * Set http://phab.localhost/config/edit/maniphest.points/ to the same as production (otherwise the data creation script will fail)
 * Set up (at least some) WMF code customizations:
 * sudo mkdir /var/www/html/phab/libext/
 * cd /var/www/html/phab/libext
 * sudo git clone https://phabricator.wikimedia.org/diffusion/PHEX/phab-extensions.git misc
 * git clone https://phabricator.wikimedia.org/diffusion/PHES/phabricator-security.git security
 * sudo /var/www/html/phab/phabricator/bin/config set load-libraries '["/var/www/html/phab/libext/misc","/var/www/html/phab/libext/security/src"]' (this content will get added in the file /var/www/html/phab/phabricator/conf/local/local.json )
 * Basically see upstream docs or Mukunda's comments though I didn't fully get it I'm afraid
 * Check that the same Priority values are in place as in WMF Phabricator via http://phab.localhost/config/edit/maniphest.priorities/; e.g. rename "Wishlist" priority field value to "Lowest".
 * Set up some custom forms:
 * Set policy.allow-public to Allow Public Visibility (otherwise we cannot set Visible To in forms to Public )
 * All forms require fiddling with "Edit Form Configuration" and "Change Default Values" and also "Change Field Order".
 * Create basics of the custom Security task form and the custom NDA protected task form by creating them via http://phab.localhost/transactions/editengine/maniphest.task/edit/form/default/ (as these forms are shown as defaults in the WMF Favorites dropdown)
 * In the video, do NOT show using the Security task form or the WMF-NDA form because way too complicated to set up locally with shown project tags and projects. Maybe another time.
 * Make sure that the default task form 2 is NOT marked for editing, so only Form 4 is used for editing. Otherwise we could not change the status of tasks via the dropdown.
 * Go to default form http://phab.localhost/transactions/editengine/maniphest.task/edit/2/ and change the form name from "New Task" to "Create Task".
 * Create a form dedicated for editing. Make that Edit Form 4 show exactly the same fields and field order as in https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/3/
 * (See the list of all WMF Forms if you're too bored).
 * Right now there is still no way to change the Priority value in tasks. Hence: Go to http://phab.localhost/transactions/editengine/maniphest.task/ and create a second form. Click "Mark as Edit Form". Do not click "Mark as Create Form". Then go to "Lock/Hide fields" and sync with https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/3/, e.g. hide Security field etc.
 * Set up Favorites menu:
 * Run /var/www/html/phab/phabricator/bin/config set phabricator.show-prototypes true (because otherwise we cannot add "Browse Events" as a link in the Favorites menu because Calendar is a prototype application)
 * Copy WMF Favorites menu settings to http://phab.localhost/favorites/menu/configure/global/
 * Remove unused Applications: Go to http://phab.localhost.org/applications and configure->uninstall Phrequent (so you don't see "Start Tracking Time" in task sidebar), Nuance, Phriction, ChatLog, Conpherence (so there won't be an icon in the top bar), Diviner, Drydock, Fund, Guides, Phortune, Phragment, Releeph.
 * UI buttons: Set http://phab.localhost/config/edit/phabricator.serious-business/ to false
 * Set up global sidebar from production to http://phab.localhost/home/menu/configure/global/
 * Set up front dashboard's panels (because needed for dashboard): Copy W727, W6, W3, to http://phab.localhost/dashboard/panel/edit/. Add W3 text panel to W6 tab panel. Ignore the other panels down there - shrug.
 * Set up front dashboard (because needed for sidebar): Copy dashboard from production to http://phab.localhost/dashboard/edit/form/default/, then use Add Dashboard to Menu ?? Add to Home Page Menu ?? Add to Global Home Menu, after that go to http://phab.localhost/home/menu/configure/global/ and move this dashboard to the very top to make it the default view
 * Create test data:
 * Create user accounts: Create six standard user accounts (as Users can not be created via the API) on http://phab.localhost/people/create/:
 * DB fiddling: mysql -u ak and then:
 * Get rid of that "Email Not Verified" in front of user names via SQL command: UPDATE phabricator_user.user u SET isEmailVerified = "1" WHERE (u.realName = "Finley" OR u.userName = "gerritbot");
 * Get the PHID-USER-xxxxxxxxxxxxxxxxxxxx of the user named Finley; then fake the "MediaWiki User" link on the user profile: INSERT INTO phabricator_user.user_externalaccount (id, phid, userPHID, accountType, accountDomain, accountID, dateCreated, dateModified, username, emailVerified, accountURI,properties,providerConfigPHID) VALUES ('1', 'PHID-XUSR-abcdefghijklmnopqrst', 'PHID-USER-xxxxxxxxxxxxxxxxxxxx', 'mediawiki', 'mediawiki', '15764393', '1410977885', '1410977885', 'Finley', '0', ' https://www.mediawiki.org:/w/index.php?title=User:Finley', '{}', 'PHID-AUTH-lzggdu7sabdjmwyrzigm');
 * Install jq package ; needed for the script to create projects and tasks
 * Start daemons via /var/www/html/phab/phabricator/bin/phd start - needed for search result indexing (and maybe also for creating test data)
 * As of August 2020, edit Phabricator's PHP code to work around https://secure.phabricator.com/T13553 (see test data creation script warning output at beginning)
 * Log in as Maria via /var/www/html/phab/phabricator/bin/auth recover Maria, then go to Settings > Conduit API Tokens, then "Generate Token"
 * Create test data: Use token from previous step to run shell script at https://gitlab.com/aklapper/phabricator-lorem-ipsum/-/blob/master/phabvideoscriptdata.sh to create projects and tasks (some with assignees and priorities), add folks as members to projects, etc.
 * If things went wrong: After screwing up creating data in the Phab test instance, run the following SQL command to remove data from the maniphest and project databases:
 * Log in as @gerritbot via /var/www/html/phab/phabricator/bin/auth recover Gerritbot and then go to task "Repair leaking pipe in bathroom" and create a fake Gerrit comment: Change 12345678 had a related patch set uploaded (by Cassidy; owner: Cassidy): [house/bath@main] Fix memory leak in bathroom pipe (`1a2b3c4de`)  https://gerrit.wikimedia.org/r/12345678  Then also add the project tag #Patch-For-Review. Then log out.
 * Log in as Cassidy, then go to http://phab.localhost/tag/plumbers, click "Workboard" in sidebar, then "Create workboard", then in upper right corner click "Manage > Add Column", then create column "In Progress". Then on the workboard, move the task "Repair leaking pipe in bathroom" into that column. (Because it is not supported to set up workboard columns via API.)
 * Log in as Cassidy, then manually make "Repair leaking pipe in bathroom" a subtask of "Take a bath". (I'm too stupid for the API call to get an ID of an existing task.)
 * Log in as admin, then go to http://phab.localhost/herald/create/, select "Maniphest Tasks", select "Global Rule", set "When all of Title contains grumpy", then every time Add Projects #ACME".
 * Log in as Finley, then go to http://phab.localhost/p/Finley/, click "Manage", click "Edit Profile". Set "Title" to "Chief wizard in training". Set Blurb to  Chief wizard; see [my mediawiki.org user profile]( https://www.mediawiki.org/w/index.php?title=User:Finley ) for more info.
 * Log in as Finley, then go to "Deploy a grumpy dog" task T6 and add a comment "Do we have any preferences on which kind of dog?" to trigger the Herald rule.
 * Go to "Food-Plants" subproject, create a workboard for it, and then go to "Manage > Edit Menu" and make the Project Details the default view (click the pin).
 * Log in as Finley, then go to "Grow potatoes" task, click "Flag For Later" in side bar, select red flag and add comment "Too yummy to forget!", then set flag (so we have something already in the Flag list)
 * Video and audio production:
 * Record audio script (with a better microphone on a silent night) as wav
 * Use post-recording noise reduction (Audacity etc.) if needed over audio files
 * Sort out width of browser window for screencasts. I wanted 16x9 format but some in-between Firefox update changed the titlebar UI element behavior and height which I did not notice, so I ended up with the strange 1675x980px ratio (instead of 1675x942px which I originally wanted, as 942÷9?16 = 1675), so start browser via firefox --width 1675 and move resulting browser window to upper left corner.
 * Start daemons via /var/www/html/phab/phabricator/bin/phd start (needed for search result indexing in the video)
 * Under GNOME Settings, go to Universal Access, and set Seeing > Cursor Size to Large, and set Pointing & Clicking > Locate Pointer to On (to show an effect when pressing the Ctrl key).
 * Make screencasts in GNOME not stop after 30 seconds:
 * Make sure to log into Wayland because the pointer stuff plus text highlighting via Ctrl is so broken under X.org
 * Increase zoom level on local Phabricator instance and mediawiki.org and phabricator.wikimedia.org in the browser to 150%
 * Log into Phabricator as Finley
 * Record the actual screencast video parts and save files as videoXX-YY-raw.webm, while listening to the audio files in parallel.
 * If needed, adjust audio files to screencast video parts by adding silence breaks etc.
 * Resize/crop area of screencast video parts:
 * Create static video parts (intro slide; end slide; page screenshots):
 * Create PNG screenshot files of web pages shown in the vide, in format 1675x980px (see above)
 * Create custom slide screenshots as SVG files in Inkscape (File > New from Template > Video) with size 1675x980px (see above), for beginning and end slides of videos
 * Convert SVG files to PNG files:
 * Convert the PNG files to videos (the -t value is the video length in seconds):
 * Make sure each audio part and corresponding video part have pretty much the same length - if there are huge differences, then fix manually. You can list the lenghts of all webm and wav files via
 * Merge each video part and audio part into one -av file (no reencoding to keep quality; no audio ripdown yet to keep quality until last rendering when we convert audio to mono etc):
 * Concatenate video parts into each tutorial video without reencoding:
 * Create text file videoXX-filelist.txt which must include lines in the format of: file videoXX-YY-av.webm
 * ffmpeg -f concat -i videoXX-filelist.txt -c:v copy ./videoXX-rc.webm
 * Final export: Use VP9 codec via two-pass, play with bitrate and quality; convert audio to mono. See https://trac.ffmpeg.org/wiki/Encode/VP9#twopass. Afterwards check resulting videos; in some cases things can go wrong and slides become unreadable:
 * Full quality:
 * Reduced quality and dimensions:
 * Subtitle production: Add time markers to srt subtitle files via gnome-subtitles or Aegisub application
 * Upload resulting videos and their corresponding subtitle files to https://commons.wikimedia.org/wiki/Category:Phabricator_Tutorial_Series