GitLab/Workflows/Security patches

The following is a proposed Gitlab workflow for managing security/embargoed patches for Wikimedia code. Note that for now, much of the workflow will likely remain the same as the current process, with the addition of using some of Gitlab's features to hopefully improve upon capabilities and efficiency.

Reporting, creating and deploying security patches

 * 1) Upon discovery of any security issues, please review the information and steps within Wikimedia's Reporting Security Bugs documentation.
 * 2) If you have opted to create a Phabricator security report (as opposed to emailing a report to [mailto:security@wikimedia.org security@wikimedia.org]), you will first need to create a Phabricator account to do so if you do not have one.  Once logged into Phabricator, perform a brief search to see if the security issue has already been reported.  If you find no evidence that it has, please submit a secure report via this form.  Submit as much detailed information as possible following the guidelines within the aforementioned Reporting Security Bugs documentation.
 * 3) An important reminder: do not post security reports or related patches to gerrit, github or any other public developer tools, websites, mailing lists, IRC channels, etc. prior to any responsible disclosure of the underlying security issue.
 * 4) For code which canonically exists within gerrit, please follow the steps below:
 * 5) Develop your patch locally (preferably on a temporary branch) and test it within MediaWiki-Vagrant, MediaWiki-Docker or an an-hoc local development environment.  It is recommended to not use a public-facing tool such as patchdemo.
 * 6) Create a patch file within your working directory by running   (where "Txxxxx" is the Phabricator task id).
 * 7) Upload the patch by attaching it to the relevant Phabricator task as a secure file attachment.  Coordinate with other developers to review your patch by reaching out to them via email, IRC, etc. and then subscribing them to the security task.  Add the Phabricator #Patch-For-Review project to the task.
 * 8) Again, an important reminder: do not upload the patch publicly to gerrit for review at this time, unless approved to do so by a member of the Security Team.
 * 9) For code which canonically exists within Gitlab, please follow the steps below:
 * 10) If you do not currently have a Wikimedia Gitlab account, you will need to create one under the register tab on the sign-in page.
 * 11) Once signed into Gitlab with your account, click on the Groups > Explore Groups option and then click the New Group button.
 * 12) Create a new group named   where   is the Phabricator task id of the task you created to track this security issue.  Ensure that the visibility of this group is Private.  Under the Members section of the group, invite relevant members of the Security Team and others who might assist with code review.  Only add as many users as necessary - the fewer users the better.
 * 13) Next, browse the list of projects and their repositories to find those relevant to the security issue.
 * 14) Once you have found and navigated to the appropriate project, click on the fork icon located in the top-right corner.
 * 15) By default, Gitlab will want to fork the repository into your user's namespace.  Instead, select the private   group listed below the pre-selected user namespace.  This may take Gitlab some time to complete depending upon the size of the repository.  To verify the new fork, examine the url in your web browser - it should contain the path:.
 * 16) Now clone the project locally and work to develop a security patch as you normally would within MediaWiki-Vagrant, MediaWiki-Docker or an an-hoc local development environment.  You'll likely want to add any relevant ssh keys to Gitlab so you can easily push changes to this newly-forked repository.  You can create a new development branch if you'd like, but since this is a forked project under a private group namespace, you can simply work from the main branch.
 * 17) When you are satisfied with your changes, notify those who will participate within the code review on the relevant Phabricator task that the forked repository is ready for review.
 * 18) Once code review has been completed, and any relevant CI has run, a patch file should be created for production deployment via the   command.
 * 19) Once the security patch has been deployed to production (see steps below), a merge request should be created to merge all relevant changes from the security issue project fork back to the original project's repository.  This should involve a merge from a main branch to a main branch.  This can be accomplished by choosing the Merge Requests menu item within the right navigation and completing the steps.  You should compare the changes and select the option to squash commits when the request is accepted.  For the merge request title and/or description,   should suffice.  This process can also be used to backport to any relevant release branches in addition to the main branch.
 * 20) An important reminder: do not create or attempt to merge a merge request until the security issue has been properly disclosed, which a member of the Security Team should be able to verify.
 * 21) Once the merge request has been approved and merged, the forked project and private group should be deleted.  For the project, this can be accomplished by navigating to Settings > Advanced > Expand > Delete project.  For the private group, this can be accomplished by navigating to Groups > Your Groups > the relevant private group > Settings > Advanced > Remove group.
 * 22) For code which canonically exists within Github, please follow the steps within Wikimedia's Reporting Security Bugs documentation and do not create any public pull requests.
 * 23) If the relevant code repositories exist canonically within another versioning system or repository, please contact [mailto:security@wikimedia.org security@wikimedia.org] for additional guidance.
 * 24) Deploying security patches to production
 * 25) Once the security issue has been privately discussed within Phabricator and all relevant security patches have been tested, reviewed and approved, they should be uploaded to the current deployment server.
 * 26) * Check that the patch applies with  on the relevant deployment branch(es).
 * 27) * Apply the patch with  to the relevant deployment branch(es).
 * Note that some config files are made public via noc.wikimedia.org; don't put anything non-public within those.
 * 1) Deploy as usual but use   to prevent the automatic logging from revealing too much information regarding affected files or components.  Log the deployment manually by saying "!log Deployed patch for Txxxxx" in.
 * 2) Ensure the security patch will be applied to future deployment branches:
 * 3) * The .patch file should be stored on the deployment host under /srv/patches/ /, for whichever branches have the security patch applied.
 * 4) ** Patches to MediaWiki core should be stored in /srv/patches/ /core/
 * 5) ** Patches to extensions should be stored under /srv/patches/ /extensions/ExtensionName.
 * 6) *** Filenames should be prefixed with a 2-digit number to indicate the order in which patches should be applied in the repo. Your file should be prefixed with '01-' if it is the first patch in the directory, or the next highest number if other patches already exist.
 * 7) *** Patch files should be git-committed to the local repository within /srv/patches/.
 * 8) * Add a note or link to a log entry to the relevant Phabricator task noting that the security patch has been deployed.
 * 9) * If the Security Team isn't already aware of this particular security issue, be sure to inform them that you deployed the patch to prevent duplicate effort.
 * 10) Work with the Security Team to make sure the vulnerability is resolved and that your patch makes it into the next security release.
 * 11) If there is no PII or other sensitive data within the associated Phabricator task, it can now be made public.  If you are unsure how to do this or if the task should be made public, please consult a member of the Security Team.
 * 12) Perform any necessary backports within the relevant versioning systems (gerrit, Gitlab, etc.) to supported release branches assuming the patch is relevant for and applies to previous versions.  As an extra step, once the branch is backported to , it can be removed from /srv/patches/ as it will no longer apply to future production release branches.  This will make Release Engineering's life a little easier.
 * 13) Request a CVE, if appropriate.  Note the CVE ID within the task's title, description and within any related security release tasks.

Potential deployment problem 1: Submodule security patches not committed
Sometimes you may find a security patch for an extension that has not been committed:

This is normal! The reason for this state is that subsequent updates to the submodules require human intervention to fix merge/rebase conflicts.