MediaWiki

User:JHernandez (WMF)/Test Html to Wikitext

< User:JHernandez (WMF)

Converted from https://gitlab.com/aklapper/phabricator-lorem-ipsum/-/raw/master/how-to-phab-tutorial-videos.html

List of steps performed in 2020 to set up a local Phabricator installation on a local Fedora machine and make it look close enough to the production instance and to make it have some data, with the goal to create Wikimedia Phabricator tutorial videos. See https://phabricator.wikimedia.org/T214522 for more context.

  • First follow the installation guide which basically means:
    • sudo dnf install httpd mariadb-server php php-gd php-mbstring php-mysqlnd php-opcache php-pecl-apcu-bc php-pecl-zip php-xml
    • sudo dnf install php-process (needed for running the daemons which say "posix PHP extension not installed")
    • sudo systemctl restart httpd.service
    • sudo systemctl restart mariadb.service
    • Open browser and go to http://localhost/ to check if httpd is up
    • sudo mkdir /var/www/html/phab/
    • cd /var/www/html/phab/
    • sudo git clone https://github.com/phacility/libphutil.git
    • sudo git clone https://github.com/phacility/arcanist.git
    • sudo git clone https://github.com/phacility/phabricator.git
  • Now follow the configuration guide:
    • On this machine, my user account is called 'ak'.
    • sudo vim /etc/httpd/conf/httpd.conf: <VirtualHost *> ServerName phab.localhost DocumentRoot /var/www/html/phab/phabricator/webroot <Directory "/var/www/html/phab/phabricator/webroot"> Options Indexes FollowSymLinks AllowOverride All Require all granted </Directory> RewriteEngine on RewriteRule ^/rsrc/(.*) - [L,QSA] RewriteRule ^/favicon.ico - [L,QSA] RewriteRule ^/(.*)$ /index.php?__path__=/$1 [B,L,QSA] ErrorLog /var/log/httpd/error_log </VirtualHost>
    • sudo mysql
    • mysql> CREATE USER 'ak'@'localhost.foo' IDENTIFIED BY '';
    • mysql> GRANT ALL PRIVILEGES ON *.* TO 'ak'@'localhost.foo' IDENTIFIED BY '';
    • sudo /var/www/html/phab/phabricator/bin/config set mysql.host "localhost"
    • sudo /var/www/html/phab/phabricator/bin/config set mysql.user ak
    • sudo /var/www/html/phab/phabricator/bin/config set mysql.pass ""
    • sudo systemctl restart mariadb.service
    • sudo /var/www/html/phab/phabricator/bin/storage upgrade --user "ak" --password "" --force
    • sudo /var/www/html/phab/phabricator/bin/config set phabricator.base-uri "http://phab.localhost"
    • sudo systemctl restart mariadb.service
    • sudo systemctl restart httpd.service
  • In /etc/php.ini change post_max_size = 8M to post_max_size = 32M
  • Go to http://phab.localhost/ and log in as admin
  • Implement WMF customizations:
  • Create test data:
    • Create user accounts: Create six standard user accounts (as Users can not be created via the API) on http://phab.localhost/people/create/:
      Maria / Maria / maria@example.com, Kate / Kate / kate@example.com, Yves / Yves / yves@example.com, Cassidy / Cassidy / cassidy@example.com, Finley / Finley / finley@example.com, and for the sake of faking it also as a standard user gerritbot / Gerrit / gerritbot@example.com.
      (See Configuring accounts how to log into these accounts from a console, basically: /var/www/html/phab/phabricator/bin/auth recover someUsername and then use the shown link.)
      Log in as Gerritbot, go to its profile, and set the user profile image to https://phab.wmfusercontent.org/file/data/ctemipegegt3vq2bjkqs/PHID-FILE-x4lhwocmrnntw7ph3wwp/profile.
    • DB fiddling: mysql -u ak and then:
      • Get rid of that "Email Not Verified" in front of user names via SQL command: UPDATE phabricator_user.user u SET isEmailVerified = "1" WHERE (u.realName = "Finley" OR u.userName = "gerritbot");
      • Get the PHID-USER-xxxxxxxxxxxxxxxxxxxx of the user named Finley; then fake the "MediaWiki User" link on the user profile: INSERT INTO phabricator_user.user_externalaccount (id, phid, userPHID, accountType, accountDomain, accountID, dateCreated, dateModified, username, emailVerified, accountURI,properties,providerConfigPHID) VALUES ('1', 'PHID-XUSR-abcdefghijklmnopqrst', 'PHID-USER-xxxxxxxxxxxxxxxxxxxx', 'mediawiki', 'mediawiki', '15764393', '1410977885', '1410977885', 'Finley', '0', 'https://www.mediawiki.org:/w/index.php?title=User:Finley', '{}', 'PHID-AUTH-lzggdu7sabdjmwyrzigm');
    • Install jq package; needed for the script to create projects and tasks
    • Start daemons via /var/www/html/phab/phabricator/bin/phd start - needed for search result indexing (and maybe also for creating test data)
    • As of August 2020, edit Phabricator's PHP code to work around https://secure.phabricator.com/T13553 (see test data creation script warning output at beginning)
    • Log in as Maria via /var/www/html/phab/phabricator/bin/auth recover Maria, then go to Settings > Conduit API Tokens, then "Generate Token"
    • Create test data: Use token from previous step to run shell script at https://gitlab.com/aklapper/phabricator-lorem-ipsum/-/blob/master/phabvideoscriptdata.sh to create projects and tasks (some with assignees and priorities), add folks as members to projects, etc.
      • If things went wrong: After screwing up creating data in the Phab test instance, run the following SQL command to remove data from the maniphest and project databases:
        START TRANSACTION; use phabricator_project; TRUNCATE TABLE edge; TRUNCATE TABLE edgedata; TRUNCATE TABLE project; TRUNCATE TABLE project_column; TRUNCATE TABLE project_columnposition; TRUNCATE TABLE project_columntransaction; TRUNCATE TABLE project_customfieldnumericindex; TRUNCATE TABLE project_customfieldstorage; TRUNCATE TABLE project_customfieldstringindex; TRUNCATE TABLE project_datasourcetoken; TRUNCATE TABLE project_project_fdocument; TRUNCATE TABLE project_project_ffield; TRUNCATE TABLE project_project_fngrams; TRUNCATE TABLE project_project_fngrams_common; TRUNCATE TABLE project_slug; TRUNCATE TABLE project_transaction; TRUNCATE TABLE project_trigger; TRUNCATE TABLE project_triggertransaction; TRUNCATE TABLE project_triggerusage; use phabricator_maniphest; TRUNCATE TABLE edge; TRUNCATE TABLE edgedata; TRUNCATE TABLE maniphest_customfieldnumericindex; TRUNCATE TABLE maniphest_customfieldstorage; TRUNCATE TABLE maniphest_customfieldstringindex; TRUNCATE TABLE maniphest_nameindex; TRUNCATE TABLE maniphest_task; TRUNCATE TABLE maniphest_task_fdocument; TRUNCATE TABLE maniphest_task_ffield; TRUNCATE TABLE maniphest_task_fngrams; TRUNCATE TABLE maniphest_task_fngrams_common; TRUNCATE TABLE maniphest_transaction; TRUNCATE TABLE maniphest_transaction_comment; use phabricator_user; TRUNCATE TABLE edge; TRUNCATE TABLE edgedata; TRUNCATE TABLE phabricator_session; COMMIT;
    • Log in as @gerritbot via /var/www/html/phab/phabricator/bin/auth recover Gerritbot and then go to task "Repair leaking pipe in bathroom" and create a fake Gerrit comment: Change 12345678 had a related patch set uploaded (by Cassidy; owner: Cassidy): [house/bath@main] Fix memory leak in bathroom pipe (`1a2b3c4de`) https://gerrit.wikimedia.org/r/12345678 Then also add the project tag #Patch-For-Review. Then log out.
    • Log in as Cassidy, then go to http://phab.localhost/tag/plumbers, click "Workboard" in sidebar, then "Create workboard", then in upper right corner click "Manage > Add Column", then create column "In Progress". Then on the workboard, move the task "Repair leaking pipe in bathroom" into that column. (Because it is not supported to set up workboard columns via API.)
    • Log in as Cassidy, then manually make "Repair leaking pipe in bathroom" a subtask of "Take a bath". (I'm too stupid for the API call to get an ID of an existing task.)
    • Log in as admin, then go to http://phab.localhost/herald/create/, select "Maniphest Tasks", select "Global Rule", set "When all of Title contains grumpy", then every time Add Projects #ACME".
    • Log in as Finley, then go to http://phab.localhost/p/Finley/, click "Manage", click "Edit Profile". Set "Title" to "Chief wizard in training". Set Blurb to Chief wizard; see [my mediawiki.org user profile](https://www.mediawiki.org/w/index.php?title=User:Finley) for more info.
    • Log in as Finley, then go to "Deploy a grumpy dog" task T6 and add a comment "Do we have any preferences on which kind of dog?" to trigger the Herald rule.
    • Go to "Food-Plants" subproject, create a workboard for it, and then go to "Manage > Edit Menu" and make the Project Details the default view (click the pin).
    • Log in as Finley, then go to "Grow potatoes" task, click "Flag For Later" in side bar, select red flag and add comment "Too yummy to forget!", then set flag (so we have something already in the Flag list)
  • Video and audio production:
    • Record audio script (with a better microphone on a silent night) as wav
    • Use post-recording noise reduction (Audacity etc.) if needed over audio files
    • Sort out width of browser window for screencasts. I wanted 16x9 format but some in-between Firefox update changed the titlebar UI element behavior and height which I did not notice, so I ended up with the strange 1675x980px ratio (instead of 1675x942px which I originally wanted, as 942÷9?16 = 1675), so start browser via firefox --width 1675 and move resulting browser window to upper left corner.
    • Start daemons via /var/www/html/phab/phabricator/bin/phd start (needed for search result indexing in the video)
    • Under GNOME Settings, go to Universal Access, and set Seeing > Cursor Size to Large, and set Pointing & Clicking > Locate Pointer to On (to show an effect when pressing the Ctrl key).
    • Make screencasts in GNOME not stop after 30 seconds:
      gsettings set org.gnome.settings-daemon.plugins.media-keys max-screencast-length "uint32 600"
    • Make sure to log into Wayland because the pointer stuff plus text highlighting via Ctrl is so broken under X.org
    • Increase zoom level on local Phabricator instance and mediawiki.org and phabricator.wikimedia.org in the browser to 150%
    • Log into Phabricator as Finley
    • Record the actual screencast video parts and save files as videoXX-YY-raw.webm, while listening to the audio files in parallel.
      • If needed, adjust audio files to screencast video parts by adding silence breaks etc.
    • Resize/crop area of screencast video parts:
      for file in videoXX-YY-raw.webm; do ffmpeg -i "$file" -b:v 2M -filter:v "crop=1675:980:0:100" "${file/%-raw.webm/-cropped.webm}"; done
    • Create static video parts (intro slide; end slide; page screenshots):
      • Create PNG screenshot files of web pages shown in the vide, in format 1675x980px (see above)
      • Create custom slide screenshots as SVG files in Inkscape (File > New from Template > Video) with size 1675x980px (see above), for beginning and end slides of videos
      • Convert SVG files to PNG files:
        for file in *.svg; do convert ${file} ${file}.png; done
      • Convert the PNG files to videos (the -t value is the video length in seconds):
        ffmpeg -loop 1 -i slide.png -c:v libvpx-vp9 -b:v 2M -t 7 -strict experimental -shortest videoXX-YY-cropped.webm
    • Make sure each audio part and corresponding video part have pretty much the same length - if there are huge differences, then fix manually. You can list the lenghts of all webm and wav files via
      for file in video*.w*; do echo "$file"; ffmpeg -i "$file" 2>&1 | grep Duration; done
    • Merge each video part and audio part into one -av file (no reencoding to keep quality; no audio ripdown yet to keep quality until last rendering when we convert audio to mono etc):
      for file in video*-cropped.wav; do ffmpeg -i "${file/%.wav/-cropped.webm}" -i "$file" -c:v copy -c:a opus -strict experimental "${file/%.wav/-av.webm}"; done
    • Concatenate video parts into each tutorial video without reencoding:
      • Create text file videoXX-filelist.txt which must include lines in the format of: file videoXX-YY-av.webm
      • ffmpeg -f concat -i videoXX-filelist.txt -c:v copy ./videoXX-rc.webm
    • Final export: Use VP9 codec via two-pass, play with bitrate and quality; convert audio to mono. See https://trac.ffmpeg.org/wiki/Encode/VP9#twopass. Afterwards check resulting videos; in some cases things can go wrong and slides become unreadable:
      • Full quality:
        ffmpeg -y -i videoXX-rc.webm -c:v libvpx-vp9 -pass 1 -b:v 1000K -threads 1 -speed 4 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -an -f webm /dev/null
        ffmpeg -i videoXX-rc.webm -c:v libvpx-vp9 -pass 2 -b:v 1000K -threads 1 -speed 0 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -c:a libopus -b:a 64k -ac 1 -f webm videoXX-final-full.webm
      • Reduced quality and dimensions:
        ffmpeg -y -i videoXX-rc.webm -c:v libvpx-vp9 -pass 1 -b:v 500K -threads 1 -speed 4 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -an -f webm /dev/null
        ffmpeg -i videoXX-rc.webm -filter:v scale=1060:-1 -c:v libvpx-vp9 -pass 2 -b:v 500K -threads 1 -speed 0 -tile-columns 0 -frame-parallel 0 -auto-alt-ref 1 -lag-in-frames 25 -g 9999 -aq-mode 0 -c:a libopus -b:a 64k -ac 1 -f webm videoXX-final-reduced.webm
  • Subtitle production: Add time markers to srt subtitle files via gnome-subtitles or Aegisub application
  • Upload resulting videos and their corresponding subtitle files to https://commons.wikimedia.org/wiki/Category:Phabricator_Tutorial_Series
Retrieved from "https://www.mediawiki.org/w/index.php?title=User:JHernandez_(WMF)/Test_Html_to_Wikitext&oldid=4094577"