-
Suggestion
-
Resolution: Won't Fix
Hello,
SSH multiplexing with bitbucket was disabled and causes many problems - especially in case you use multiple repositories at once (this is for example problem when you do android AOSP development and use repo and gerrit - both of them are often performing operations on many repos at the same time. For repo, it is not a big problem - just a warning message. For gerrit, it is a huge problem as replication fails permanently when SSH multiplexing cannot be established. There are some nasty workarounds in place,
The change happened in July and it is plaguing us till today (we are using some workarounds after we found the reason but It would be much better if it is solved by enabling mulitplexing back - because the workarounds put unnecessary pressure on your infrastructure - for example now we have a job that mirrors about 100 repositories to bitbucket every 10 minutes and that would not be necessary at all if ssh multiplexing is working.
More details can be found in https://answers.atlassian.com/questions/39449158/answers/43549761/comments/43550008
And I was redirected from https://support.atlassian.com/servicedesk/customer/portal/11/BBS-42335 to create the issue.
Let me copy some parts of this that are relevant:
"Up until a couple of days ago (I guess before the LFS beta was available), I was able to repo sync my android tree with bitbucket, and had no issues replicating with gerrit to bitbucket. Ever since that change, I get this error when repo syncing:
channel 2: open failed: administratively prohibited: cannot open additional channels
mux_client_request_session: session request failed: Session open refused by peer
And something very similar with gerrit replication. I tried temporarily switching my repos to another git server provider, and as I expected everything is ok. Any info on this? Thanks."
Aug 12:
"I struggled with this for a while but had no time to take a closer look. Now I know what happens and I have a workaround.
The problem is that Bitbucket disabled capability of multiplexing for SSH connections. In short: multiplexing works in the way that you only connect to the same server once. All the other connections are executed as additional channels opened via the single master connection already opened. Git automatically runs ssh commands in multiplexing mode under the hood. You can see that if you run the following while repo sync is running:
ps -eaf | grep ssh
You will see something similar to:
#!text ssh -o ControlMaster=no -o ControlPath=some_path_here .....
Unfortunately you cannot use .ssh/config (because -o parameters override it).
UPDATE: See below answer - apparently you can have ControlMaster set to yes in .ssh and it works fine
The solution that I found was to change default SSH command that GIT uses and disable multiplexing altogether. You can set GIT_SSH_COMMAND as follows:
#!text
export GIT_SSH_COMMAND="ssh -o ControlPath=none"
Once you do it - git will use your ssh command instead of the default and it will open new SSH connection for every git command separately. It's a bit slower than when multiplexing works but when Bitbucket rejects multiplexed channels, repo sync is even slower (by default ssh will fall-back to non-multiplexing connection when the multiplexing fails so it still works despite the error messages - but much slower).
Add it to your .bashrc or similar and you should be good to go.
One drawback of this solution is that it is a global setting - all your git via ssh will stop using multiplexing. "
Oct 19:
Update: It seems also that works (and it localised to Bitbucket only - so it's better solution):
In your .ssh/config add ControlMaster yes for bitbucket:
Host bitbucket.org
ControlMaster yes
Michael Bayer -> we also had problem with replication. Some repos from Gerrit failed to replicate to Bitbucket (especially when there were branches or deleted changes). I know it's a lame solution but we solved it in a simple way. Additionally to built-in Gerrit replication we also run a crontab job every 10 minutes to mirror our repos to Bitbucket - so most of the changes are replicated almost immediately by Gerrit, but those that fail are later re-synced using the external mirroring...
Here is the script that we use:
#!bash #!/bin/bash date pushd /mnt/gerrit/git SKIPPED_REPOS="(All-Projects.git|All-Users.git)" for repo in * do echo if [[ $repo =~ ^$SKIPPED_REPOS$ ]]; then echo "Skipping mirroring $repo" continue fi if [ ! -d "$repo" ]; then echo "Skipping non-directory $repo" continue fi pushd $repo >/dev/null REPO_URL=git@bitbucket.org:someprefix/$repo echo "Mirroring $repo to $REPO_URL" git push --mirror $REPO_URL 2>&1 popd >/dev/null done popd date
It mirrors a bit more than the default gerrit replication (includes gerrit config as well for the repo) - but it's still OK.
It also increases traffic for Bitbucket quite significantly. I wonder if the latest SSH outages are connected to the fact that some people like us are generating lot more traffic for Atlassian than is needed because we are workarounding some not-well-thought changes on their side.
Hey Atlassian !!! - Hear us out please. Enable the SSH mulitplexing back. It will save you ton of traffic and infrastructure to handle it.
[BCLOUD-13465] Please enable SSH multiplexing back
dt0b35, I upgraded to Eclipse 2021-03 (4.19.0) and now the SSH client is broken when connecting to Bitbucket. The interesting thing is that they have defaulted to Apache Mina which is what I had selected before they deprecated JSch.
It is not the same failure as you and gives "Cannot log in at bitbucket.org:22". I suspect this is related to my private key which is 2048 bit RSA. When I try to open the key in Preferences->General->Network Connections->SSH2->Key Management it fails to load because only 1024 bit is supported and I suspect this may be related to the problem. Unfortunately many organizations require keys stronger than 1024 and so downgrading won't always work.
Hi Chad,
in eclipse Version: 2021-03 (4.19.0) Build id: 20210312-0638 there no longer is a SSH selection box above the http client.
I guess the JSch was removed and Apache Mina became the default (and only one)
Additionally it does not happen for all Bitbucket repositories but just for some (common criteria I did not find).
Locally I will clone via console and for eclipse I created a bug-report: https://bugs.eclipse.org/bugs/show_bug.cgi?id=573205
dt0b35: What SSH client is configured in your Git preferences? Are you using Apache MINA sshd? I am cloning via SSH using Eclipse 2020-12 (4.18.0) and it works.
Hi Chad,
thanks for the hint but (on my eclipse?) it looks like the "Java Built-in HTTP" is Apache as well.
After switching in preferences still
Caused by: org.apache.sshd.common.channel.exception.SshChannelOpenException: cannot open additional channels at org.apache.sshd.client.channel.AbstractClientChannel.handleOpenFailure(AbstractClientChannel.java:388)
appears when try to clone a repo from Bitbucket (even after restarting eclipse).
I am cloning via SSH and so the HTTP Client will not change the implementation 
In response to dt0b35: I figured out how to make this work.
Bitbucket breaks the functionality of Apache HTTP client while Eclipse is fetching from git. As a workaround you can change the client to Java Built-in HTTP in Preferences->Version Control->Git->HTTP Client.
ControlMaster yes
in the .ssh/config for bitbucket.org does not help for eclipse Version: 2021-03 (4.19.0) Build id: 20210312-0638 with (default) jgit. (windows).
... Caused by: org.apache.sshd.common.SshException: DefaultOpenFuture[ChannelExec[id=1, recipient=-1]-JGitClientSession[git@bitbucket.org/104.192.141.1:22]]: Failed (SshChannelOpenException) to execute: cannot open additional channels at org.apache.sshd.common.future.AbstractSshFuture.lambda$verifyResult$1(AbstractSshFuture.java:131) at org.apache.sshd.common.future.AbstractSshFuture.formatExceptionMessage(AbstractSshFuture.java:185) ... Caused by: org.apache.sshd.common.channel.exception.SshChannelOpenException: cannot open additional channels at org.apache.sshd.client.channel.AbstractClientChannel.handleOpenFailure(AbstractClientChannel.java:388) at org.apache.sshd.common.session.helpers.AbstractConnectionService.channelOpenFailure(AbstractConnectionService.java:577) at org.apache.sshd.common.session.helpers.AbstractConnectionService.process(AbstractConnectionService.java:467) at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:502) ...
Any workarounds/solutions for eclipse?
(git clone on the command prompt works as expected)
It looks like this breaks git clone from Eclipse using Apache SSHD with jgit.
Caused by: java.io.IOException: git-upload-pack 'gisfederal/diu-ingest-realtime.git' timed out after 30 seconds while opening the channel at org.eclipse.jgit.transport.sshd.SshdSession.exec(SshdSession.java:307) at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:254) ... 13 more Caused by: org.apache.sshd.common.channel.exception.SshChannelOpenException: cannot open additional channels
Note that the "timed out after 30 seconds" message is incorrect because the failure is immediate. Instead it failed with "cannot open additional channels". I am not sure how to configure the eclipse client to disable multiplexing. Any thoughts?
@shamruk Have you tried the --jobs argument to parallelize the submodules? Meanwhile, it's interesting that you get a 5-6x boost from multiplexing. Connecting to SSH should not require that much time. At what time (UTC) do you typically interact with Bitbucket via SSH?
@jjurisoo Some sites discourage the use of SSH for this and similar reasons; and others – honestly they simply haven't gotten here yet in terms of server management. As far as improving performance overall, we're currently working on improving git performance for all users with the addition of Mercurial-style clone bundles. See: http://public-inbox.org/git/CADoxLGPFgF7W4XJzt0X+xFJDoN6RmfFGx_96MO9GPSSOjDK0EQ@mail.gmail.com/
There is quite a speed difference with the same repo on Bitbucket versus that other big Git site, but I don't know what other factors are involved or how much multiplexing would help. In addition, it is annoying having to make the exception for Bitbucket when you work with multiple machines with multiple OS installations.
Anything that can be done to help speed up response time and reduce configuration steps would sway me to continue to subscribe to Bitbucket services.
I do understand the issue of visibility, but I would (rhetorically) ask the question about how other sites get away with it.
When I make a recursive fetch (about 8 submodules) multiplex speeds up the process 5-6 times. Any workaround to speedup it without multiplex?
Unfortunately, we do not plan to add multiplexing capabilities into our new stack. One of the reasons for not support multiplexing in our new designs was to allow us better control over resource allocation and traffic shaping for SSH requests. By allowing a single connection to open multiple channels on a target node we lose visibility and control at the point of ingress.
I realize that this may create a complicated situation for some systems that assume multiplexing is available, however, as there are valid workarounds, it's worth it to us to keep Bitbucket healthy and give us visibility into what is happening on our systems.
With 4k RSA authentication keys on a smartcard, it takes 1-2 seconds to generate signatures. Support for the SSH Master functionality would reduce this computation to the initial connection, but Bitbucket has issues with it. (It works fine with Github, Gerrit, etc., only Bitbucket is broken.)
Another issue that the lack of multiplexing creates is that it makes recursive clones very slow and annoying. Our projects often have one or more submodules. Some projects have as many as 6 submodules. With Git's default behavior of enabling multiplexing, a single entry of the SSH key's password authenticates all subsequent connections to recursively check out all of the repos.
When Bitbucket disabled multiplexing, the key password must be supplied for the initial repo and all submodules. This also means a new handshake to initiate a new connection for each clone, adding around a 5 second delay when cloning each submodule. This wasted time really starts to add up if you have to clone a large number of repos with many submodules throughout the day.
We've added a modification very similar to what Guillaume posted above to our SSH configs. This removes a small amount of the latency and removes the annoying "cannot open additional channels" error messages for each clone.
Our best solution to avoid having to authenticate the key repeatedly is to use an SSH agent, for example:
eval `ssh-agent -s` && ssh-add -t 60s
There are other agents, but this one is typically available without installing anything extra.
We had to change our local config files to reuse sockets.
Host * !bitbucket.org ControlMaster auto ControlPath /tmp/ssh-%r_%h:%p.ctl ControlPersist 30
Correct Dato. But this does not work with gerrit which uses Jsch Java ssh client internally (and it does not read that configuration option).
FYI you can configure ~/.ssh/config to skip certain hosts:
Host !bitbucket.org
ControlMaster auto
ControlPath ~/.ssh/...
Any update on this? We are facing the same issues with gerrit replication and using crontab workaround from Jarek. Setting the .ssh/config ControlMaster yes didn't seem to help
Hi @Chad Juliano,
your "Cannot log in" failure is another known issue with RSA keys in eclipse 2021-03 with default EGit 5.11:
RSA SSH Key does not work for this EGit version. (see https://bugs.eclipse.org/bugs/show_bug.cgi?id=572056 )
You need to create a new up-to-date ed25519 key: How to setup SSH in order to work with Bitbucket (see https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54 )
cd %userprofile%\.ssh
ssh-keygen -t ed25519 -f YOUR_USER_ID-bitbucket-ed25519
and update your %userprofile%\.ssh\config the IdentityFile for Host bitbucket.org and add it to https://bitbucket.org/account/settings/ssh-keys/
(at least there is no reason to go back to the old RSA key when JGit is fixed
)