Gerrit/Troubleshooting

This is the central place to document troubleshooting any problems with Git, Gerrit and git-review.

fatal: The remote end hung up unexpectedly
See https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-unexpectedly-while-git-cloning for some ideas, and debug the problem by setting the  environment variable.

git-review complains about problems encountered installing commit-msg hook
If you encounter this error when trying to push changes using git-review, you are not working with a repository that was cloned via ssh. You must clone repositories using ssh, not http or https, to successfully push changes using git-review.

git-review complains about "missing Change-id in commit message"
If you forgot to run, "remote" will complain about "missing Change-id in commit message".

But it will also suggest a commit message with a Change-Id: INNNXXXNNN... line.

Either: Copy that line starting with "Change-Id", run, and paste the Change-Id line under your commit message in the text editor that opens up. Or it will suggest a hook fix:   You should be able to use either method (but the hook didn't work for me), then repeat  and it should complete.

git-review complains "You have more than one commit that you are about to submit"
If  displays a warning about multiple commits, followed by a list of other people's commits that have already been merged, perform the following workaround:

This will ensure git-review has an up-to-date view of the remote master.
 * 1) Respond "no" to abort git-review.
 * 2) Run   or   (Both commands do exactly the same thing, they fetch objects from all remote repositories set. So just pick the command you remember easily and forget about the other one.)
 * 3) Rerun



Background
This was fixed in 2012 but the bug returned. Wikimedia projects suffer from this disproportionately because  is set on most Wikimedia projects, and the bug only triggers when rebasing is disabled (using either that setting or the   flag).

git-review complains "Cannot query patchset information"
git-review does not work correctly if git generates non-English output. You will see an error like this:

$ git review -d 62474 Cannot query patchset information The following command failed with exit code 255 "ssh -x -p None gerrit query --format=JSON --current-patch-set change:62474" --- Bad port ' None' ---

This is due to a bug in git-review.

To work around this on a Linux system, either apply the patch from the bug report above, or set up an alias that forces git to use English output. To do so, put this into your  or similar setup file:

alias git="LANG=C git"

git-review complains "Could not parse json query response: u'Verified'"
git-review version 1.18 has been reported to have issues when trying to review a change from Gerrit. You will see an error like this:

$ git review -d 76352 Could not parse json query response: u'Verified'

This seams to be due to a bug in git-review version 1.18 since version 1.12 and version 1.21 work correct.

To work around this on a Linux system, use another version like version 1.12 on Fedora or version 1.12, version 1.21 on Ubuntu (by downgrading or removing version 1.18 and installing the suitable rpm). Version 1.22 under fedora works also.

git-review complains "ConfigParser.NoSectionError: No section: 'updates'"
If this happens (T57732, fixed in newer git-review versions):

$ git review -s Traceback (most recent call last): File "/usr/local/bin/git-review", line 1196, in    main File "/usr/local/bin/git-review", line 1108, in main needs_update = latest_is_newer File "/usr/local/bin/git-review", line 179, in latest_is_newer if not configParser.getboolean("updates", "check"): File "/usr/lib/python2.7/ConfigParser.py", line 368, in getboolean v = self.get(section, option) File "/usr/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'updates'

Adding something like this to ".gitreview" fixes the problem:

git-review doesn't like merge commits
If you merged a development branch, and now want to submit a merge commit to Gerrit,  may not let you. It may ask you for submitting lots of changes from one of the merged branches, or otherwise mangle the commit. To avoid this, push the commit directly to Gerrit, bypassing :

git push gerrit HEAD:refs/for/master

For more information, see the Gerrit documentation.

git-review complains "Working tree is dirty"
If upon doing 'git review' you receive a message "Working tree is dirty" try doing 'git add' for the file(s) changed (or created), then, and then. (This was seen on Mac OS X with an older git client.)

git-review complains about a missing Change-Id in the commit message
If upon doing 'git review' you receive a message about 'missing Change-Id', then your /.git/hooks/commit-msg is probably incorrect. It should look something like:

CHANGE_ID_AFTER="Bug|Issue" MSG="$1"

add_ChangeId { clean_message=`sed -e ' /^diff --git a\/.*/{ s/// q               } /^Signed-off-by:/d /^#/d ' "$MSG" | git stripspace` if test -z "$clean_message" then return fi
 * 1) Check for, and add if missing, a unique Change-Id

You will also get a missing Change-ID message when trying to merge some change from git that does not have. It seems that the hook isn't called by cherry-pick, but it is fortunately called by.

In the example below we will be moving trivial change bd7a268e4be2da23ba0b9943c3b0ba1ac88294dc from git master in the mediawiki/core project (it came via SVN trunk) to REL1_19.

Sample session to exhibit this problem:

(...full trivial diff to one file omitted...)

To fix this, use  (don't commit) option to  : git cherry-pick -n bd7a268e4be2da23ba0b9943c3b0ba1ac88294dc git commit -c bd7a268e4be2da23ba0b9943c3b0ba1ac88294dc

Unfortunately, if you want to add the original commit ID to the message (as done by ) you have to add it yourself.

The change above has been submitted as 3c88c61f1b7e36d5d374a42bb0f50783ab5391a4 for  review.

Gerrit complains that "Your change could not be merged due to a path conflict"
You have to rebase your change in order to be able to merge it.

For the impatient:

 git checkout master git pull git-review -d &lt;change #> git rebase origin/master git status &lt;edit "both modified" files in status report> git add &lt;files> git rebase --continue git review

Full explanation


So you have submitted a change and was approved. But by the time it gets reviewed other commits have altered the main repository which now cause a conflict. Gerrit is unable to automatically merge your change into the repository, you will have to fix it and resubmit the change.

The example below is based on a real use case : change 2514 using the operations/puppet repository

First fetch the change using  and its   option :

(production)$ git-review -d 2514 Downloading refs/changes/14/2514/1 from gerrit into review/hashar/ignore_pyc Switched to branch 'review/hashar/ignore_pyc' (review/hashar/ignore_pyc)$

is the user name,  the topic name he gave. Notice how git-review automatically placed you to the branch.

You now have to rebase on top of the main branch. The change on gerrit shows the branch, just add "gerrit/" in front. For this change in the operations/puppet repo, the main branch is "production", so rebase on gerrit/production; for other repos it's usually.

(review/hashar/ignore_pyc)$ git rebase gerrit/production First, rewinding head to replay your work on top of it... Applying: pyc files are now ignored Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging .gitignore CONFLICT (content): Merge conflict in .gitignore Failed to merge in the changes. Patch failed at 0001 pyc files are now ignored When you have resolved this problem run "git rebase --continue". If you would prefer to skip this patch, instead run "git rebase --skip". To check out the original branch and stop rebasing run "git rebase --abort". (no branch)$

The key in the above output is the line starting with CONFLICT. It tells you the filename that git couldn't cleanly rebase, usually because your patch alters it and changes in the main branch have also altered it. Running git status confirms it:

(no branch)$ git status (no branch)$
 * 1) Not currently on any branch.
 * 2) Unmerged paths:
 * 3)   (use "git reset HEAD ..." to unstage)
 * 4)   (use "git add/rm ..." as appropriate to mark resolution)
 * 5) 	both modified:     .gitignore
 * 1) 	both modified:     .gitignore

Edit the conflicting file (in this case .gitignore). This will have,     markers surrounding the conflicting lines, you must clean this up. During the merge conflict git creates,   , and    files with the three source versions. You can use a three-way merge tool to pick which lines to use; git mergetool wraps the use of a merge tool.

Once you have finished editing, you have to add that modification to have it used during the rebase then continue fixing any conflicting patches:

(no branch)$ git add .gitignore (no branch)$ git rebase --continue Applying: pyc files are now ignored (review/hashar/ignore_pyc)$

Since there was no more patches to fix, you have been placed back in the review/hashar/ignore_pyc branch. Looking at the log:

$ git log -n5 --decorate --pretty=oneline ...
 * a3631d2 (HEAD, review/hashar/ignore_pyc) pyc files are now ignored (2 seconds ago
 * 1b6cd67 (gerrit/production, production) ensure sample config file removed (18 hours ago)

Verify you change looks fine before resubmitting it to gerrit. Just use git show &lt;sha1 of commit>, i.e. . You can eventually amend it to state you have rebased the change.

Now submit your change back in the repository:

(review/hashar/ignore_pyc)$ git review remote: Resolving deltas:  0% (0/2) To ssh:// @gerrit.wikimedia.org:29418/operations/puppet.git (review/hashar/ignore_pyc)$

Heading back to Gerrit, the change is a new patchset pending review.

Congratulations on fixing your first rebase / merge conflict!

In Gerrit your change is not merged after receiving a +2
If someone enters +2 Code Reviewed, it should trigger a series of automated builds and tests. Continuous integration/Workflow describes the flow.

In Gerrit, the "user" jenkins-bot should add a comment
 * Starting gate-and-submit jobs. https://integration.wikimedia.org/zuul/

Follow the link to see the progress of the tests for your change that Zuul has submitted to Jenkins.

If you don't see the "Starting gate-and-submit jobs" comment in gerrit, look at https://integration.wikimedia.org/zuul/ It shows everything that Zuul has submitted to Jenkins, and the "Queue length" number on top is the number of events not yet processed. If your job doesn't appear below this on the page below but the queue length is non-zero, it is likely in the queue; Zuul and Jenkins are probably busy and you just have to wait. But if the queue length is 0 and your change doesn't appear, it means it isn't going to happen and you need to resubmit it. Note that new repos have to be manually added to jenkins-bot's list.

If the required tests all pass (note that some are "non-voting"), then jenkins-bot will add a comment "Build succeeded" followed by "Change has been successfully merged into the git repository."

In Gerrit Jenkins gave a -2 so you need to override Jenkins
First of all, you should more or less never do this. (So far the reason was backporting a change to an old release that had broken tests). If Jenkins has a -2 you generally can't merge the change. What you need to do:
 * Removing the jenkins-bot "Verified" review (click the )
 * Review the latest patch set with "Verified +2" in addition to "Code-Review +2" (this is normally reserved for Jenkins, but since we're overriding it we're replacing its score with our own)
 * Click "Publish and Submit"

fatal: Could not read from remote repository.
If you get the error Then please share the output of the command, run the affected git command with debug logging variables such as, and test if   shows

fatal: The remote end hung up unexpectedly
If you get the error Then you're not logged in to your ssh key right now. Solution: do  to make it prompt you for the passphrase for your key and add it to the active keychain. Then you can check what's in your keychain with. Then try pushing for review again.

The fingerprint of the Gerrit server is

dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a:2e:bf:92:e1

so you can say yes when it asks you to add that fingerprint to the  file.

Keep in mind that gerrit listens on port 29418 and if for some reason you forgot to specify port number, you might be hitting "normal" SSH daemon listening on port 22 (this one has RSA key fingerprint b5:e9:dc:b2:dd:6e:70:f7:18:8a:dc:a3:5d:ab:99:4d).

To check whether SSH connectivity and public key authentication, use  which should show

Your change is not having all tests run
Not all tests are run for all users. See Continuous integration/Whitelist for more information.

Push using HTTPS (when SSH is not functional)
This method is helpful for submitting to changes to Gerrit when SSH is not functional (for example SSH is blocked by your institution or internet provider).

If SSH is not working, you should see one of these error messages: ssh: connect to host gerrit.wikimedia.org port 29418: Connection refused ssh: connect to host gerrit.wikimedia.org port 29418: Network is unreachable

You can also explicitly try the command ssh -p 29418 your-user-name@gerrit.wikimedia.org If SSH is functional, that command should create the following output: Connection to gerrit.wikimedia.org closed by remote host. Connection to gerrit.wikimedia.org closed.

When SSH is not functional, you need an HTTP(S) Password which can be generated in the Account Settings of Gerrit under "HTTP Password". After generating the password, commit all the changes for your patch(s) and use git push https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/core HEAD:refs/for/master The authentication credentials need to be entered to successfully submit the changes. The url given above is for MediaWiki core and will vary for extensions accordingly. For example if you want to push to Extension:LiquidThreads the command would be git push https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/extensions/LiquidThreads HEAD:refs/for/master

To always use https, initially clone with: git clone https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/core

Or adjust an existing repository with: git remote set-url origin https://&lt;username>@gerrit.wikimedia.org/r/mediawiki/core

Then you can use  normally.

You can use git-credential-store to store your HTTPS password so you don't have to type it in every time.

Commit Hook and Change-ID
A major problem that arises when using HTTPS for submitting changes is that commit hook is not automatically attached. A hack for this approach is to make one fail attempt to push. On doing so, the error message will automatically highlight the Change-Id, see below example: Username for 'https://gerrit.wikimedia.org': xxxxxx Password for 'https://xxxxxx@gerrit.wikimedia.org': Counting objects: 25, done. Delta compression using up to 2 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 448 bytes | 0 bytes/s, done. Total 4 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3) remote: Processing changes: refs: 1, done remote: ERROR: missing Change-Id in commit message footer remote: Suggestion for commit message: remote: Commit message appears here remote: remote: Change-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx remote: remote: Hint: To automatically insert Change-Id, install the hook: remote:  gitdir=$(git rev-parse --git-dir); scp -p -P 29418 xxxxxx@gerrit.wikimedia.org:hooks/commit-msg ${gitdir}/hooks/ remote: remote: To https://gerrit.wikimedia.org/r/mediawiki/extensions/Test ! [remote rejected] HEAD -> refs/for/master (missing Change-Id in commit message footer) error: failed to push some refs to Now copy the change ID and amend your commit. Always add Change-ID as the last line of your commit message. git commit --amend This will open an editor to change the commit message. Paste the Change-Id as the last line of the message and save it. See example: Your commit summary Your commit message Bug: Txxxxx Change-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Now you can successfully push to Gerrit.

Automatically adding commit Hook
To obtain the commit-msg script: scp -p -P 29418 USERNAME@gerrit.wikimedia.org:hooks/commit-msg /.git/hooks/ See https://gerrit.wikimedia.org/r/Documentation/cmd-hook-commit-msg.html for details.

Being behind a proxy server
If you are behind a proxy server, you also need to clone over HTTPS. Make sure the environment variable  is set correctly. To debug issues, try running with, e.g.

In case of issues, try providing  directly, e.g.

or use git's options directly:

This last option should also work for SOCKS proxy servers, using

Push using SSH only (when HTTPS is disabled not functional)
Add to your local '.gitconfig' following section: [url "ssh://username@gerrit.wikimedia.org:29418/"] pushInsteadOf = https://gerrit.wikimedia.org/r/ pushInsteadOf = https://gerrit.wikimedia.org/r/
 * 1) Use SSH over HTTPS

"git commit --amend" complains "you are in the middle of a merge -- cannot amend"
When after rebasing and merging your git commit --amend

results in message: fatal: You are in the middle of a merge -- cannot amend.

apply these steps and reapply your changes git stash git reset --hard git checkout master git review -d git stash pop git commit -a --amend If, after  jenkins-bot emails 'This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset. 'This might mean that server master branch now has merge conflicts with your patch. Check advanced Gerrit usage to see how to fix them.

"Your change requires a recursive merge to resolve"
If you get the error "Your change requires a recursive merge to resolve", you need to rebase the change set against master.
 * 1) Make sure your master branch is up to date:
 * 2) Create and switch to a new branch in which to checkout the change set with the conflict:
 * 3) Checkout the conflicting change set into this branch. You can copy/paste the correct command from the 'Download' section in the Gerrit review. It will look something like this:
 * 4) Rebase against master:
 * 5) Push the change to Gerrit for review:
 * 6) Re-review the change set in Gerrit, and then submit the changes to be merged to master.

"[remote rejected] master -> master" and "failed to push some refs"
If push to a branch other than, you will receive something along the lines of:

This means you tried to commit to branch "master" instead of submitting your changes for review.

The following is a similar error where  was trying to push to a non-existing remote branch:

You can see with  what's the branch you're currently using. Try rebasing to push correctly:

"[remote rejected] HEAD -> refs/publish/master/??? (Cannot merge change to another branch)
When attempting to cherry-pick a change or merge an entire branch and then submit for review; gerrit will be very confused if you do not explicitly tell it what branch you're modifying. You might get errors like:

! [remote rejected] HEAD -> refs/publish/master/mwalker_test (change 59546 closed)

or

! [remote rejected] HEAD -> refs/publish/master/mwalker_test (no new changes)

In this case you must ensure that your branch's  file has the correct defaultbranch option (e.g. it should be pointing to the branch you're attempting to modify) or you must manually push the refspec using

! [remote rejected] HEAD -> refs/changes/79/550379/2 (cannot add patch set to 550379.)
This is happens if you are not the original author (owner) of a change (in this case 550379).

Committer email address does not match your user account.
remote: ERROR: committer email address (email) remote: ERROR: does not match your user account.

There are two possible problems that could cause this error. If the email address that git pops back _is_ the email address you intend to use with Gerrit, then you should add that email address in Gerrit, and make sure you click the confirmation link in the email Gerrit sends you. Then try pushing again.

If, however, git sends back some nonsense email (like one you don't use anymore, or a local mail address like root@localhost), you should do the following:

To do it locally for just that repo, instead do:

Then, try pushing again.

Build failed due to merge conflict
After `git review` jenkins-bot emails 'This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset.' This might mean that server master branch now has merge conflicts with your patch.


 * Make sure your master branch hard-reset to the gerrit's master (to avoid git review complaining about multiple commits)


 * Checkout for review, rebase, recommit

If the error occurs even though you have rebased your patch as above, try making an arbitrary change to your commit message, and then running  again (related to ).

"Everything up-to-date" message after git push
If you attempt to do 'git push' after doing 'git commit' you may receive a response 'Everything up-to-date'. You have not pushed to the branch. You have to do 'git review' to move your changes to gerrit, and only from gerrit will the branch be updated. This seems to be a side effect of checking out master as a branch as of February 2012.

In some projects (e.g. test/) it is possible to do 'git push' instead of 'git review' and have the push succeed. It is probably better not to do that, as it confuses those who find your changes later and don't know where they came from.

Tagging an extension doesn't work
If you want to tag your extension, you need to connect to gerrit using ssh, not https. In your .git/config, the remote url should look like this:

To tag an extension, use git tag followed by git push :