Uploaded image for project: 'Bitbucket Cloud'
  1. Bitbucket Cloud
  2. BCLOUD-13465

Please enable SSH multiplexing back

    XMLWordPrintable

    Details

    • Feedback Policy:

      Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      Description

      Hello,

      SSH multiplexing with bitbucket was disabled and causes many problems - especially in case you use multiple repositories at once (this is for example problem when you do android AOSP development and use repo and gerrit - both of them are often performing operations on many repos at the same time. For repo, it is not a big problem - just a warning message. For gerrit, it is a huge problem as replication fails permanently when SSH multiplexing cannot be established. There are some nasty workarounds in place,

      The change happened in July and it is plaguing us till today (we are using some workarounds after we found the reason but It would be much better if it is solved by enabling mulitplexing back - because the workarounds put unnecessary pressure on your infrastructure - for example now we have a job that mirrors about 100 repositories to bitbucket every 10 minutes and that would not be necessary at all if ssh multiplexing is working.

      More details can be found in https://answers.atlassian.com/questions/39449158/answers/43549761/comments/43550008

      And I was redirected from https://support.atlassian.com/servicedesk/customer/portal/11/BBS-42335 to create the issue.

      Let me copy some parts of this that are relevant:

      "Up until a couple of days ago (I guess before the LFS beta was available), I was able to repo sync my android tree with bitbucket, and had no issues replicating with gerrit to bitbucket. Ever since that change, I get this error when repo syncing:
      channel 2: open failed: administratively prohibited: cannot open additional channels
      mux_client_request_session: session request failed: Session open refused by peer
      And something very similar with gerrit replication. I tried temporarily switching my repos to another git server provider, and as I expected everything is ok. Any info on this? Thanks."

      Aug 12:
      "I struggled with this for a while but had no time to take a closer look. Now I know what happens and I have a workaround.
      The problem is that Bitbucket disabled capability of multiplexing for SSH connections. In short: multiplexing works in the way that you only connect to the same server once. All the other connections are executed as additional channels opened via the single master connection already opened. Git automatically runs ssh commands in multiplexing mode under the hood. You can see that if you run the following while repo sync is running:
      ps -eaf | grep ssh
      You will see something similar to:

      #!text
      
      ssh -o ControlMaster=no -o ControlPath=some_path_here .....
      

      Unfortunately you cannot use .ssh/config (because -o parameters override it).
      UPDATE: See below answer - apparently you can have ControlMaster set to yes in .ssh and it works fine
      The solution that I found was to change default SSH command that GIT uses and disable multiplexing altogether. You can set GIT_SSH_COMMAND as follows:

      #!text
      
      export GIT_SSH_COMMAND="ssh -o ControlPath=none"
      
      

      Once you do it - git will use your ssh command instead of the default and it will open new SSH connection for every git command separately. It's a bit slower than when multiplexing works but when Bitbucket rejects multiplexed channels, repo sync is even slower (by default ssh will fall-back to non-multiplexing connection when the multiplexing fails so it still works despite the error messages - but much slower).
      Add it to your .bashrc or similar and you should be good to go.
      One drawback of this solution is that it is a global setting - all your git via ssh will stop using multiplexing. "

      Oct 19:
      Update: It seems also that works (and it localised to Bitbucket only - so it's better solution):
      In your .ssh/config add ControlMaster yes for bitbucket:
      Host bitbucket.org
      ControlMaster yes
      Michael Bayer -> we also had problem with replication. Some repos from Gerrit failed to replicate to Bitbucket (especially when there were branches or deleted changes). I know it's a lame solution but we solved it in a simple way. Additionally to built-in Gerrit replication we also run a crontab job every 10 minutes to mirror our repos to Bitbucket - so most of the changes are replicated almost immediately by Gerrit, but those that fail are later re-synced using the external mirroring...
      Here is the script that we use:

      #!bash
      
      #!/bin/bash
      date
      pushd /mnt/gerrit/git
      SKIPPED_REPOS="(All-Projects.git|All-Users.git)"
      for repo in *
      do
      echo
      if [[ $repo =~ ^$SKIPPED_REPOS$ ]]; then
      echo "Skipping mirroring $repo"
      continue
      fi
      if [ ! -d "$repo" ]; then
      echo "Skipping non-directory $repo"
      continue
      fi
      pushd $repo >/dev/null
      REPO_URL=git@bitbucket.org:someprefix/$repo
      echo "Mirroring $repo to $REPO_URL"
      git push --mirror $REPO_URL 2>&1
      popd >/dev/null
      done
      popd
      date
      
      

      It mirrors a bit more than the default gerrit replication (includes gerrit config as well for the repo) - but it's still OK.
      It also increases traffic for Bitbucket quite significantly. I wonder if the latest SSH outages are connected to the fact that some people like us are generating lot more traffic for Atlassian than is needed because we are workarounding some not-well-thought changes on their side.
      Hey Atlassian !!! - Hear us out please. Enable the SSH mulitplexing back. It will save you ton of traffic and infrastructure to handle it.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            jarek1297140092 potiuk
            Votes:
            6 Vote for this issue
            Watchers:
            9 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: