Please enable SSH multiplexing back

Hello,

SSH multiplexing with bitbucket was disabled and causes many problems - especially in case you use multiple repositories at once (this is for example problem when you do android AOSP development and use repo and gerrit - both of them are often performing operations on many repos at the same time. For repo, it is not a big problem - just a warning message. For gerrit, it is a huge problem as replication fails permanently when SSH multiplexing cannot be established. There are some nasty workarounds in place,

The change happened in July and it is plaguing us till today (we are using some workarounds after we found the reason but It would be much better if it is solved by enabling mulitplexing back - because the workarounds put unnecessary pressure on your infrastructure - for example now we have a job that mirrors about 100 repositories to bitbucket every 10 minutes and that would not be necessary at all if ssh multiplexing is working.

More details can be found in https://answers.atlassian.com/questions/39449158/answers/43549761/comments/43550008

And I was redirected from https://support.atlassian.com/servicedesk/customer/portal/11/BBS-42335 to create the issue.

Let me copy some parts of this that are relevant:

"Up until a couple of days ago (I guess before the LFS beta was available), I was able to repo sync my android tree with bitbucket, and had no issues replicating with gerrit to bitbucket. Ever since that change, I get this error when repo syncing:
channel 2: open failed: administratively prohibited: cannot open additional channels
mux_client_request_session: session request failed: Session open refused by peer
And something very similar with gerrit replication. I tried temporarily switching my repos to another git server provider, and as I expected everything is ok. Any info on this? Thanks."

Aug 12:
"I struggled with this for a while but had no time to take a closer look. Now I know what happens and I have a workaround.
The problem is that Bitbucket disabled capability of multiplexing for SSH connections. In short: multiplexing works in the way that you only connect to the same server once. All the other connections are executed as additional channels opened via the single master connection already opened. Git automatically runs ssh commands in multiplexing mode under the hood. You can see that if you run the following while repo sync is running:
ps -eaf | grep ssh
You will see something similar to:

#!text

ssh -o ControlMaster=no -o ControlPath=some_path_here .....

Unfortunately you cannot use .ssh/config (because -o parameters override it).
UPDATE: See below answer - apparently you can have ControlMaster set to yes in .ssh and it works fine
The solution that I found was to change default SSH command that GIT uses and disable multiplexing altogether. You can set GIT_SSH_COMMAND as follows:

#!text

export GIT_SSH_COMMAND="ssh -o ControlPath=none"

Once you do it - git will use your ssh command instead of the default and it will open new SSH connection for every git command separately. It's a bit slower than when multiplexing works but when Bitbucket rejects multiplexed channels, repo sync is even slower (by default ssh will fall-back to non-multiplexing connection when the multiplexing fails so it still works despite the error messages - but much slower).
Add it to your .bashrc or similar and you should be good to go.
One drawback of this solution is that it is a global setting - all your git via ssh will stop using multiplexing. "

Oct 19:
Update: It seems also that works (and it localised to Bitbucket only - so it's better solution):
In your .ssh/config add ControlMaster yes for bitbucket:
Host bitbucket.org
ControlMaster yes
Michael Bayer -> we also had problem with replication. Some repos from Gerrit failed to replicate to Bitbucket (especially when there were branches or deleted changes). I know it's a lame solution but we solved it in a simple way. Additionally to built-in Gerrit replication we also run a crontab job every 10 minutes to mirror our repos to Bitbucket - so most of the changes are replicated almost immediately by Gerrit, but those that fail are later re-synced using the external mirroring...
Here is the script that we use:

#!bash

#!/bin/bash
date
pushd /mnt/gerrit/git
SKIPPED_REPOS="(All-Projects.git|All-Users.git)"
for repo in *
do
echo
if [[ $repo =~ ^$SKIPPED_REPOS$ ]]; then
echo "Skipping mirroring $repo"
continue
fi
if [ ! -d "$repo" ]; then
echo "Skipping non-directory $repo"
continue
fi
pushd $repo >/dev/null
REPO_URL=git@bitbucket.org:someprefix/$repo
echo "Mirroring $repo to $REPO_URL"
git push --mirror $REPO_URL 2>&1
popd >/dev/null
done
popd
date

It mirrors a bit more than the default gerrit replication (includes gerrit config as well for the repo) - but it's still OK.
It also increases traffic for Bitbucket quite significantly. I wonder if the latest SSH outages are connected to the fact that some people like us are generating lot more traffic for Atlassian than is needed because we are workarounding some not-well-thought changes on their side.
Hey Atlassian !!! - Hear us out please. Enable the SSH mulitplexing back. It will save you ton of traffic and infrastructure to handle it.