-
Suggestion
-
Resolution: Won't Fix
-
Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.
It would be nice to see support for the LargeFiles extension with Mercurial 2.1+, even if it is only for paid plans
[BCLOUD-3843] Mercurial - LargeFiles support (BB-3903)
I am considering three solutions:
- self-hosted HG repository based on RhodeCode Enterprise. It is free for teams up to 25 - https://rhodecode.com/get-started
- HG on Helix TeamHub with mirror for OSS projects on SourceForge.net. Perforce has a strong history and also free for teams up to 5
- HG on sourcehut - feels big potential
Due to Mercurial nature, it is easy to migrate source code from one solution to another (for luck, I have used BB only for code hosting).
Yeah, I've checked out all of those. Unfortunately, it seems to me that there is no other Mercurial hosting option that would a) have a big, profitable and reputable (in my book Atlassian is losing this status pretty quickly which is a shame) company behind it with support for Mercurial for the foreseeable future, b) offer tools for code review at least as good as Bitbucket's and using concepts widely accepted in the software development world and c) offer a hosted solution for a price reasonable for this part of our toolbox.
No need to migrate to git if you like mercurial. Mercurial is till a greate tool, and there is other providers than Bitbucket out there: https://www.mercurial-scm.org/wiki/MercurialHosting
We're using Mercurial at our team primarily, and have been on Bitbucket for the last six years as a company, so this is a huge letdown. I’ve got 8 notifications yesterday about updates to such Bitbucket issues and my “neat, news coming!” was very quickly turned sour. Not to mention all this comes within a year of the same happening with HipChat...
However, if you want help moving to Git (which we might end up doing after advocating Mercurial for years...) check out this: https://lombiq.com/bitbucket-mercurial-to-git-migration We have some free and open source tools for converting repos or otherwise you can hire us to do the whole migration and train your team.
Atlassian has committed a terrible wrong. You bought one of the few companies that delivered first-rate Mercurial hosting and killed it. If you’re not going to provide Mercurial support, the very least you could do is release this part of Bitbucket cloud to the open-source community.
After much consideration, we’ve decided to remove Mercurial support from Bitbucket Cloud and the API. Mercurial support will be officially removed on June 1, 2020.
The decision to deprecate was not easy as we knew there were customers (like yourselves) who were currently using Mercurial for versioning their code. Ultimately we concluded that sunsetting Mercurial support will allow us to singularly focus on building the best possible experience for the lions share of the market and our users.
That said, if any critical Mercurial issues arise over the coming months, please reach out to our support team. Our goal is to continue offering the same Mercurial experience you have today, until June 1, 2020.
Read on to learn more about what led to this decision, timeline, and migration resource support.
@seanfarley the new LFS experimental extension is now included with all Mercurial 4.6+ distributions. Any chance of revisiting the issue? We have a number of LargeFile repositories that we are hoping to move to migrate to Bitbucket + LFS.
Thanks for your feedback on this ticket. We review our highest voted issues regularly, and have just completed this process again.
Unfortunately, we don't have any plans to address this feature right now. At the moment, we're busy improving Bitbucket in the following areas:
- Pull requests
- File search
- Deployments
- Integrations
Please continue to follow this ticket for updates, and add comments describing any use-cases that are important to your team and not already covered above.
Kind regards,
Martin
I'd be happy to give you my two cents if it doesn't affect anything else in my or my team's workflow. We'd use LF, though just lightly (having a few 50-100MB binaries here and there), but if it works (well) then we'll possibly move some more data to Bitbucket that is now stored elsewhere.
I'm hoping to have some time in the coming weeks to look into the lfs support that Matt added. I could use some help, though. If anyone wants to test, I suspect that pushing / pulling will work just fine. The Bitbucket UI might be a different story, though. What doesn't work exactly? What does work?
If anyone is willing to help test, that'd be great!
Will upcoming 4.5 release somehow push this issue?
#!python 1.1.4. Largefiles changes largefiles: add a 'debuglfput' command to put largefile into the store largefiles: add support for 'largefiles://' url scheme largefiles: allow to run 'debugupgraderepo' on repo with largefiles largefiles: convert EOL of hgrc before appending to bytes IO largefiles: explicitly set the source and sink types to 'hg' for lfconvert largefiles: modernize how capabilities are added to the wire protocol
Recently, Matt Harbison has added lfs support in Mercurial that can be found here:
https://www.mercurial-scm.org/repo/hg/file/tip/hgext/lfs
I'd recommend testing that out to see how it works and let me know about any problems.
Currently, I'm waiting for @facebook to finish their LFS implementation. There are two ways to help out
- help Facebook finish the LFS implementation by joining the mercurial-devel mailing list
- help finish the branch that Piotr started for last year's Google Summer of Code https://bitbucket.org/liscju/hg-largefiles-gsoc
If either of those two things work, then I'd be able to finish the task on our backend
This would be a great and much needed addition for Mercurial. Any news to share about the progress ?
@liscju made a lot of progress this summer and I'm very proud of him Unfortunately, we're in a place where I need to review and merge his code, as well as get the Bitbucket side into shape.
During the same time, @facebook started their own implementation of LFS which is actually using the same protocol. Using this might be more ideal for Bitbucket since that would mean we'd only need one implementation on the server-side of things.
Hi @ArneBab and @seanfarley,
Can you please update us on the status of this issue?
Thank you
When I was using BlueGene/P (or Q), I would just install Mercurial in my home directory. This is mainly due to it being difficult to separate out a core extension. We'll see how this summer code goes, though.
Also, I find it hard to argue that clusters need largefiles but that's another discussion. For now, I would recommend installing it into your home directory. Anyways, time for more coffee and more coding
@seanfarley If you’re an admin to the cluster, it’s no problem to update. At my institute, however, we struggle with not that responsive admins. Would it be an option to restrict the changes to the largefiles extension and provide this also as separate extension, which makes it easy for users to add BitBucket largefiles support when they can’t update Mercurial itself?
Yay GSoC!
Re git lfs install: Yikes! Somehow that wasn’t mentioned very prominently on the git lfs page <
@zdm: If GSoC works out, that would be 6 months. It feels like this is finally moving. @seanfarley already added support for changeset obsolescence, which allows using Mutable-HG with BitBucket — you can safely and collaboratively rewrite history using BitBucket as backend. I think that shows that @seanfarley can push this forwards.
@kafumanto No problem The largefiles project idea I had was accepted for this year's Summer of Code and, on top of that, it looks like there is a student willing to work on it! Here's to hoping there is some fruit to that labor _
Responding to both @arneb and @TigerC10 in this reply:
Would the Mercurial large files be stored on BitBucket similar to the git large files?
Neither Git nor Mercurial large files will ever be on our backend. They will be hosted somewhere in The Cloud™. This is by design by both us and Github so that our servers never have to incur the cost of that huge transfer.
Would people need to update Mercurial for it to work?
Yep.
It's easy enough to see if your Mercurial version is larger than 2.0, but if it's only 3 years old you should be okay. The use of the Mercurial large files options is just as easy as the github large files
...
Unlike git, largefile support is baked into Mercurial - so it should be easier to adopt and less "hacky".
As I mentioned before, the current implementation of Mercurial largefile support won't work for Bitbucket because that hardcodes the url for the largefile to be the same as the path for the code (in this case, our core Bitbucket servers). This design makes it impossible for us to offload that work (and bandwidth cost) to a dedicated datacenter for hosting large files.
Your post to the mailing list sounds like largefiles at BitBucket would require the most up to date Mercurial — which is a blocker for many people I know (we have to be able to run 3 years old Mercurial, because that’s what’s on the cluster).
I, too, have been the admin of computing clusters. And I, too, have been a user of some clusters. Even on the most painful of painful powerpc and aix machines, I could always update to the newest Mercurial since at its core, you just need a working libc.
So it would be very useful, if BitBucket could provide regular large files support as it existed in 2013.
Sure, no problem. Let me hop in my time machine
Or rather, practically put, the instructions to use largefiles should be at least as simple as those for git lfs.
Don't worry, it will. For git lfs support, you need Go (or a precompiled binary) which is very hard on a cluster. So, in that regard, this will be easier. For Mercurial, you can pip install the newest version and you'll be good-to-go.
@seanfarley Thanks Sean for having provided an official response to this long waited feature. For some projects, this feature is really important. Please, continue to keep us updated on this issue
@ArneBab: The last edit on the Largefiles Extension page on the Mercurial wiki was on 2013-09-01. In fact, the original page's creation was on 2011-10-18. The Largefiles Extension shipped with Mercurial 2.0, which released on 2011-11-01
It's easy enough to see if your Mercurial version is larger than 2.0, but if it's only 3 years old you should be okay. The use of the Mercurial large files options is just as easy as the github large files (I believe that github's LF support was inspired by Mercurial's). You can learn more from https://www.mercurial-scm.org/wiki/LargefilesExtension but suffice it to say that either passing a --large flag to your hg add command or by setting the largefiles.size or largefiles.patterns config option for your repo to automatically use the largefiles plugin.
Unlike git, largefile support is baked into Mercurial - so it should be easier to adopt and less "hacky".
@seanfarley: I know that you’re doing great work here — that we can set a repository as non-publishing has been a great step forward towards enabling the features Mercurial provides easily which are missing in Git (though I did not use those for collaboration yet).
Would the Mercurial large files be stored on BitBucket similar to the git large files? Would people need to update Mercurial for it to work?
Your post to the mailing list sounds like largefiles at BitBucket would require the most up to date Mercurial — which is a blocker for many people I know (we have to be able to run 3 years old Mercurial, because that’s what’s on the cluster). So it would be very useful, if BitBucket could provide regular large files support as it existed in 2013.
Or rather, practically put, the instructions to use largefiles should be at least as simple as those for git lfs. People in this thread are willing to pay for largefile support.
Is there a chance that BitBucket would hire more people to improve Mercurial support and reduce your backlog?
And could you give your Marketing team another prod? That deceptive Git vs. Mercurial article in the Atlassian Blogs is still unchanged, almost 4 years after its significant factual errors were called out.
@ArneBab Heh, I was actually working on a response before I got distracted by the deploy
I've already been thinking about how to integrate Mercurial's largefiles support since I've started working here, especially now that we've publicly announced LFS support.
One of the main show stoppers for Mercurial's largefile extension is that it is designed to push the largefiles to the same server as the changesets. This is no good for us since we want to avoid that transfer cost entirely. I've addressed these limitations on the Mercurial mailing list already:
http://markmail.org/message/7j4ohopxspcnpnhw
I've also gone ahead and volunteered to be a Google Summer of Code mentor:
If no one volunteers for the project, then I'll queue it up to my already long backlog. As for the Virtuos Games company, you heard about it the same time I did. I really wish they would have contacted the Mercurial community so we could have worked together
@Valentin: Note that it’s not Atlassian which got big with Mercurial. Bitbucket got big with Mercurial and was then bought by Atlassian. Also Atlassian is still spreading lies about Mercurial in the Atlassian blog by hosting a guest entry by a git zealot which is filled with factual errors, some even disproven in the examples in the article. Despite being called out on that in public, they did not even see the need to add a note to that guest entry about misunderstanding by the author.
I asked the Atlassian marketing team personally several times to correct this. I know they read it, because people I used to collaborate with work at the BitBucket Mercurial support.
Dear BitBucket, this is where you could be: Virtuos Games uses BitTorrentSync with Mercurial for game development with decentralized large asset storage.
I guess they show that there is room for a Mercurial hosting company. I‘m sorry for the great Mercurial developers working at Atlassian to improve Mercurial support. I know you’re doing great work and I hope you will prove me wrong on this. But from the outside it seems like you’re being used to hide hostility by the parent company against the core part of their own product. “…we decided to collaborate with GitHub on building a standard for large file support” — seriously? There is already a standard for large file support which has been part of Mercurial core since 2011, and works almost seamlessly. It just needs support from BitBucket to be easier for BitBucket users.
This crazyness is a new spin on never trust a company: never ever trust a zealot with a tool which helps “the other side”: They are prone to even put zeal over business. For everyone at BitBucket: If this isn’t a wakeup call, I don’t know what is.
Mercurial support has been neglected by Atlassian for quite some time now. Too sad... Atlassian seems to be on it's way to become the arrogant greedy corporation that only cares about profit, and ignores the very users that helped grow this product in first place.
So close... I don't really want to switch to Git, now that finally have trained teammates to use Mercurial. Is there any hope for LargeFiles for Mercurial users?
My team demands largefile support for Mercurial repos. It'd be very cool to have one. We will even pay for it!
I'm considering switching to bitbucket from github, some features like the ability to upvote issues make me want to. But I'd like large filte storage to be supported on here as well, like the exisiting git-annex.
+1 on this, between Mercurial's built-in largefiles extension, Github's LFS, and https://github.com/lionheart/git-bigstore something should be done about Bitbucket's lack of support for large binary assets.
Github LFS is now out of preview and generally available: https://github.com/blog/2069-git-large-file-storage-v1-0
I would be willing to pay more for this feature (already a paid team user).
+1 on this. With Github announcing support, things are looking much more attractive over there (despite per-private-repo pricing tiers)
Now github added a paid largefiles program. How about doing the same? https://github.com/blog/1986-announcing-git-large-file-storage-lfs
Mercurial can already beat that since 2011.
I actually started my current project with the assumption that bitbucket supports this... I'm surprised it doesn't. Hopefully this project is small enough that I don't have to move to hosting on my own server.
Any plans to implement this feature?
Can somebody from staff tell us, and close this issue if answer is "NO"?
If you're looking for a way to get rid of largefiles try http://stackoverflow.com/a/14453480/492
We are now using git-fat with bitbucket, though we loose reviewing of our images. See http://git.faceshift.com/git-fat for our fork of git-fat, which works on windows, linux, and mac.
Kiln is free for 1-2 users, see http://www.fogcreek.com/kiln/student-and-startup/ (sorry Atlassian but it has on your backlog for > 2-1/2 years and you're not especiallyshort of resources, really). Also, Kiln does Git & Hg with the same repo - seamless translations.
btw: +1 for Largefiles, please.
So should we switch to Kiln or are you going to support this?
Everyone, if you need hosted Mercurial with Largefiles, use Kiln, starting at $18/month for 5 users.
You would think that it could at least be made an available option for paying subscribers. It would certainly be worth it to me, and would still be a much better deal than going with that "other" file service. :o)
What is unfortunate is that if bitbucket does not get interested into looking at issues such as large file support, then other file service that seem to "get" the job done will render bitbucket obsolete.
BB would not have to do anything more but officially state here that the download section can generally be used for largefiles objects. If this is the case, it is possible to provide a modified largefiles extension that just uses that as remote store. If they do not do this, or state that it is not allowed, nobody will invest time. Unfortunately, it seems like they are not really interested in giving feedback to this discussion.
If you only need code hosting then yes, there are perfect alternatives out there. If you need at least pull requests as well then things are not so nice.
Helix TeamHub looks like a good service but I'm not confident they'll keep hg support for long. Mercurial there seems to be a second-class citizen like at Bitbucket, and [their blog|https://www.perforce.com/blog/vcs/git-vs-mercurial-how-are-they-different] is not exactly reassuring either. If I were you I'd just to with sourcehut.