Gerrit/Troubleshooting

This is the central place to document troubleshooting any problems with Git, Gerrit and git-review.

fatal: The remote end hung up unexpectedly
See https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-unexpectedly-while-git-cloning for some ideas, and debug the problem by setting the environment variable.

The authenticity of host '[gerrit.wikimedia.org]:29418 (....)' can't be established.
In this case, please check if the value given in the line " RSA key fingerprint is SHA256:j7HQoQ6fIuEgDHjONjI2CZ+2Iwxqgo2Ur5LbPqBgxOU " matches the fingerprint published on the wikitech:Help:SSH_Fingerprints/gerrit.wikimedia.org:29418.

If the values are the same, just press "yes".

If for some reason the values are not the same, there is a security risk and you stop trying to connect immediately.

scp of commit-msg hook fails with "subsystem request failed on channel 1"
Gerrit's Clone with commit-msg hook copy/paste commands include a  command that downloads the commit-msg file to the local repository. OpenSSH versions >=9 will fail as scp will attempt to use sftp since scp has been deprecated. To download the file, add a  to the command. For example,

"Please, commit your changes or stash them before you can merge."
To discard your changes (and anything you had in the stash): Now you can proceed with your pull.

git-review complains about problems encountered installing commit-msg hook
If you encounter this error when trying to push changes using git-review, you are not working with a repository that was cloned via ssh. You must clone repositories using ssh, not http or https, to successfully push changes using git-review.

git-review complains about "missing Change-id in commit message"
If upon doing 'git review' you receive a message about 'missing Change-Id', then your /.git/hooks/commit-msg is probably incorrect. It should look something like:

You will also get a missing Change-ID message when trying to merge some change from git that does not have. It seems that the hook isn't called by cherry-pick, but it is fortunately called by.

In the example below we will be moving trivial change bd7a268e4be2da23ba0b9943c3b0ba1ac88294dc from git master in the mediawiki/core project to REL1_19.

To fix this, use  (don't commit) option to  :

Unfortunately, if you want to add the original commit ID to the message (as done by ) you have to add it yourself.

If you forgot to run, "remote" will complain about "missing Change-id in commit message".

But it will also suggest a commit message with a Change-Id: INNNXXXNNN... line.

Either: You should be able to use either method (but the hook didn't work for me), then repeat, and it should complete.
 * Copy that line starting with "Change-Id", run, and paste the Change-Id line under your commit message in the text editor that opens up.
 * Or it will suggest a hook fix:

git-review complains "You have more than one commit that you are about to submit"
If  displays a warning about multiple commits, followed by a list of other people's commits that have already been merged, perform the following workaround:

This will ensure git-review has an up-to-date view of the remote master.
 * 1) Respond "no" to abort git-review.
 * 2) Run   or   (Both commands do exactly the same thing, they fetch objects from all remote repositories set. So just pick the command you remember easily and forget about the other one.)
 * 3) Rerun



Background
This was fixed in 2012 but the bug returned. Wikimedia projects suffer from this disproportionately because  is set on most Wikimedia projects, and the bug only triggers when rebasing is disabled (using either that setting or the   flag).

git-review complains "Cannot query patchset information"
git-review does not work correctly if git generates non-English output. You will see an error like this:

This is due to a bug in git-review.

To work around this on a Linux system, either apply the patch from the bug report above, or set up an alias that forces git to use English output. To do so, put this into your  or similar setup file:

git-review complains "ConfigParser.NoSectionError: No section: 'updates'"
If this happens (T57732, fixed in newer git-review versions):

Adding something like this to ".gitreview" fixes the problem:

git-review doesn't like merge commits
If you merged a development branch, and now want to submit a merge commit to Gerrit,  may not let you. It may ask you for submitting lots of changes from one of the merged branches, or otherwise mangle the commit. To avoid this, push the commit directly to Gerrit, bypassing :

For more information, see the Gerrit documentation.

git-review complains "Working tree is dirty"
If upon doing 'git review' you receive a message "Working tree is dirty" try doing 'git add' for the file(s) changed (or created), then, and then. (This was seen on Mac OS X with an older git client.)

git-review complains "Authentication failed"
If you see then you are trying to push via HTTPS instead of SSH which you had set up earlier. You likely want to configure git to use the SSH remote instead of the HTTPS remote.

Gerrit complains that "Your change could not be merged due to a path conflict"
You have to rebase your change in order to be able to merge it.

For the impatient:

git checkout master git pull git-review -d &lt;change #> git rebase origin/master git status &lt;edit "both modified" files in status report> git add &lt;files> git rebase --continue git review

Full explanation


So you have submitted a change and was approved. But by the time it gets reviewed other commits have altered the main repository which now cause a conflict. Gerrit is unable to automatically merge your change into the repository, you will have to fix it and resubmit the change.

The example below is based on a real use case : change 2514 using the operations/puppet repository

First fetch the change using  and its   option :

is the user name,  the topic name he gave. Notice how git-review automatically placed you to the branch.

You now have to rebase on top of the main branch. The change on gerrit shows the branch, just add "gerrit/" in front. For this change in the operations/puppet repo, the main branch is "production", so rebase on gerrit/production; for other repos it's usually.

The key in the above output is the line starting with CONFLICT. It tells you the filename that git couldn't cleanly rebase, usually because your patch alters it and changes in the main branch have also altered it. Running  confirms it:

Edit the conflicting file (in this case .gitignore). This will have,     markers surrounding the conflicting lines, you must clean this up. During the merge conflict git creates,   , and    files with the three source versions. You can use a three-way merge tool to pick which lines to use; wraps the use of a merge tool.

Once you have finished editing, you have to add that modification to have it used during the rebase then continue fixing any conflicting patches:

Since there was no more patches to fix, you have been placed back in the review/hashar/ignore_pyc branch. Looking at the log:

Verify you change looks fine before resubmitting it to gerrit. Just use, i.e.  . You can eventually amend it to state you have rebased the change.

Now submit your change back in the repository:

Heading back to Gerrit, the change is a new patchset pending review.

Congratulations on fixing your first rebase / merge conflict!

In Gerrit your change is not merged after receiving a +2
If someone enters +2 Code Reviewed, it should trigger a series of automated builds and tests. Continuous integration/Workflow describes the flow.

In Gerrit, the "user" jenkins-bot should add a comment
 * Starting gate-and-submit jobs. https://integration.wikimedia.org/zuul/

Follow the link to see the progress of the tests for your change that Zuul has submitted to Jenkins.

If you don't see the "Starting gate-and-submit jobs" comment in gerrit, look at https://integration.wikimedia.org/zuul/ It shows everything that Zuul has submitted to Jenkins, and the "Queue length" number on top is the number of events not yet processed. If your job doesn't appear below this on the page below but the queue length is non-zero, it is likely in the queue; Zuul and Jenkins are probably busy and you just have to wait. But if the queue length is 0 and your change doesn't appear, it means it isn't going to happen and you need to resubmit it. Note that new repos have to be manually added to jenkins-bot's list.

If the required tests all pass (note that some are "non-voting"), then jenkins-bot will add a comment "Build succeeded" followed by "Change has been successfully merged into the git repository."

In Gerrit Jenkins gave a -2 so you need to override Jenkins
First of all, you should more or less never do this. (So far the reason was backporting a change to an old release that had broken tests). If Jenkins has a -2 you generally can't merge the change. What you need to do:
 * Removing the jenkins-bot "Verified" review (click the )
 * Review the latest patch set with "Verified +2" in addition to "Code-Review +2" (this is normally reserved for Jenkins, but since we're overriding it we're replacing its score with our own)
 * Click "Publish and Submit"

"Permission denied (publickey)" error
In general you should check SSH verbose output with  (you can get more details with  /  but usually debug level 1 is enough). There are many reasons you can get this error, most of them related to local misconfiguration, like your SSH key not being readable, or not matching the one you uploaded to Gerrit, or using an outdated algorithm (see also T276486).

fatal: Could not read from remote repository.
If you get the error

OpenSSH disabled support for old  keys since version 8.8. however will continue to generate RSA keys when used without arguments. Such keys will just be ignored, resulting in the cryptic error message above. You need to either create a new  key as described in the Gerrit tutorial (recommended) or re-enable support for RSA keys with OpenSSH's   option, as described in the release notes.

Then please share the output of the command, run the affected git command with debug logging variables such as, and test if   (replace   in that command with your developer account's username) shows

fatal: The remote end hung up unexpectedly
If you get the error

Then you're not logged in to your ssh key right now. Solution: do  to make it prompt you for the passphrase for your key and add it to the active keychain. Then you can check what's in your keychain with. Then try pushing for review again.

The fingerprint of the Gerrit server is

dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a:2e:bf:92:e1

so you can say yes when it asks you to add that fingerprint to the  file.

Keep in mind that gerrit listens on port 29418 and if for some reason you forgot to specify port number, you might be hitting "normal" SSH daemon listening on port 22 (this one has RSA key fingerprint b5:e9:dc:b2:dd:6e:70:f7:18:8a:dc:a3:5d:ab:99:4d).

To check whether SSH connectivity and public key authentication, use  which should show

Your change is not having tests run
Tests are only run users in the CI allow list. See Continuous integration/Allow list for more information.

Push using HTTPS (when SSH is not functional)
This method is helpful for submitting to changes to Gerrit when SSH is not functional (for example SSH is blocked by your institution or internet provider).

If SSH is not working, you should see one of these error messages:

ssh: connect to host gerrit.wikimedia.org port 29418: Connection refused ssh: connect to host gerrit.wikimedia.org port 29418: Network is unreachable

You can also explicitly try the command

If SSH is functional, that command should create the following output:

Connection to gerrit.wikimedia.org closed by remote host. Connection to gerrit.wikimedia.org closed.

When SSH is not functional, you need an HTTP(S) Password which can be generated in the Account Settings of Gerrit under "HTTP Password". After generating the password, commit all the changes for your patch(s) and use git push https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/core HEAD:refs/for/master The authentication credentials need to be entered to successfully submit the changes. The url given above is for MediaWiki core and will vary for extensions accordingly. For example if you want to push to Extension:LiquidThreads the command would be git push https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/extensions/LiquidThreads HEAD:refs/for/master

To always use https, initially clone with: git clone https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/core

Or adjust an existing repository with: git remote set-url origin https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/core

Then you can use  normally.

You can use git-credential-store to store your HTTPS password so you don't have to type it in every time.

Commit Hook and Change-ID
A major problem that arises when using HTTPS for submitting changes is that commit hook is not automatically attached. A hack for this approach is to make one fail attempt to push. On doing so, the error message will automatically highlight the Change-Id, see below example: Username for 'https://gerrit.wikimedia.org ': xxxxxx Password for 'https://xxxxxx@gerrit.wikimedia.org ': Counting objects: 25, done. Delta compression using up to 2 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 448 bytes | 0 bytes/s, done. Total 4 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3) remote: Processing changes: refs: 1, done remote: ERROR: missing Change-Id in commit message footer remote: Suggestion for commit message: remote: Commit message appears here remote: remote: Change-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx remote: remote: Hint: To automatically insert Change-Id, install the hook: remote:  gitdir=$(git rev-parse --git-dir); scp -p -P 29418 xxxxxx@gerrit.wikimedia.org:hooks/commit-msg ${gitdir}/hooks/ remote: remote: To https://gerrit.wikimedia.org/r/mediawiki/extensions/Test ! [remote rejected] HEAD -> refs/for/master (missing Change-Id in commit message footer) error: failed to push some refs to

Now copy the change ID and amend your commit. Always add Change-ID as the last line of your commit message. git commit --amend

This will open an editor to change the commit message. Paste the Change-Id as the last line of the message and save it. See example: Your commit summary Your commit message Bug: Txxxxx Change-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Now you can successfully push to Gerrit.

Automatically adding commit Hook
To obtain the commit-msg script: See https://gerrit.wikimedia.org/r/Documentation/cmd-hook-commit-msg.html for details.

Being behind a proxy server
If you are behind a proxy server, you also need to clone over HTTPS. Make sure the environment variable  is set correctly. To debug issues, try running with, e.g.

In case of issues, try providing  directly, e.g.

or use git's options directly:

This last option should also work for SOCKS proxy servers, using

Push using SSH only (when HTTPS is disabled not functional)
Add to your local '.gitconfig' following section:

"git commit --amend" complains "you are in the middle of a merge -- cannot amend"
When after rebasing and merging your

results in

apply these steps and reapply your changes If, after  jenkins-bot emails 'This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset. 'This might mean that server master branch now has merge conflicts with your patch. Check advanced Gerrit usage to see how to fix them.

"Your change requires a recursive merge to resolve"
If you get the error "Your change requires a recursive merge to resolve", you need to rebase the change set against master.
 * 1) Make sure your master branch is up to date:
 * 2) Create and switch to a new branch in which to checkout the change set with the conflict:
 * 3) Checkout the conflicting change set into this branch. You can copy/paste the correct command from the 'Download' section in the Gerrit review. It will look something like this:
 * 4) Rebase against master:
 * 5) Push the change to Gerrit for review:
 * 6) Re-review the change set in Gerrit, and then submit the changes to be merged to master.

"[remote rejected] master -> master" and "failed to push some refs"
If push to a branch other than, you will receive something along the lines of:

This means you tried to commit to branch "master" instead of submitting your changes for review.

The following is a similar error where  was trying to push to a non-existing remote branch:

You can see with  what's the branch you're currently using. Try rebasing to push correctly:

"[remote rejected] HEAD -> refs/publish/master/??? (Cannot merge change to another branch)
When attempting to cherry-pick a change or merge an entire branch and then submit for review; gerrit will be very confused if you do not explicitly tell it what branch you're modifying. You might get errors like:

or

In this case you must ensure that your branch's  file has the correct defaultbranch option (e.g. it should be pointing to the branch you're attempting to modify) or you must manually push the refspec using

! [remote rejected] HEAD -> refs/publish/master (prohibited by Gerrit: not permitted: create)
If you use git-review to submit patches to gerrit, make sure you have 1.27 or newer (not 1.26). git-review 1.26 does not work with Gerrit 3.

! [remote rejected] HEAD -> refs/changes/79/550379/2 (cannot add patch set to 550379.)
This happens if you are not the original author (owner) of a change (in this case 550379). To be able to add a new patch set to a commit you did not author, please request addition to the Trusted-Contributors group on Gerrit. Every member of this group is able to add you on request.

Committer email address does not match your user account.
remote: ERROR: committer email address (email) remote: ERROR: does not match your user account.

There are two possible problems that could cause this error. If the email address that git pops back _is_ the email address you intend to use with Gerrit, then you should add that email address in Gerrit, and make sure you click the confirmation link in the email Gerrit sends you. Then try pushing again.

If, however, git sends back some nonsense email (like one you don't use anymore, or a local mail address like root@localhost), you should do the following:

To do it locally for just that repo, instead do:

Then, try pushing again.

Build failed due to merge conflict
After `git review` jenkins-bot emails 'This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset.' This might mean that server master branch now has merge conflicts with your patch.


 * Make sure your master branch hard-reset to the gerrit's master (to avoid git review complaining about multiple commits)


 * Checkout for review, rebase, recommit

If the error occurs even though you have rebased your patch as above, try making an arbitrary change to your commit message, and then running  again (related to ).

"Everything up-to-date" message after git push
If you attempt to do 'git push' after doing 'git commit' you may receive a response 'Everything up-to-date'. You have not pushed to the branch. You have to do 'git review' to move your changes to gerrit, and only from gerrit will the branch be updated. This seems to be a side effect of checking out master as a branch as of February 2012.

In some projects (e.g. test/) it is possible to do 'git push' instead of 'git review' and have the push succeed. It is probably better not to do that, as it confuses those who find your changes later and don't know where they came from.

Tagging an extension doesn't work
If you want to tag your extension, you need to connect to gerrit using ssh, not https. In your .git/config, the remote url should look like this:

To tag an extension, use git tag followed by git push :

Plugin install error: TypeError: self.onAction is not a function
If you see this error please hard refresh your browser cache.

For some the Ctrl + F5 or Shift + Ctrl + F5 (or whatever combination your browser wants) is not sufficient.

If you still run into the issue after pressing your browsers F5 combination, please try clearing your caches for good.

For example on Firefox follow steps 1-6 from


 * https://support.mozilla.org/kb/how-clear-firefox-cache#w_clear-the-cache

If you use a different browser, it should allow you to clear cached web content somewhere in the setting too. Please find that in your browser's help pages and follow the instructions.

Bad server host key: Invalid key length
Users who are using Fedora 33 or later with the the  or   crypto settings are unable to use Git over SSH on Gerrit. (related to T326204)

Affected users can work around this issue by decreasing the minimum key length requirement for  specifically in their   file: