We have addressed large 8GB builds with a recent change, described here: https://blog.bitbucket.org/2018/02/20/support-large-builds-bitbucket-pipelines/

I believe this edge case bug has also been fixed. Please let us know if it is still causing issues.

Issue BCLOUD-13834 was marked as a duplicate of this issue.

Thanks Matt, see BCLOUD-13874

As this is already an issue for us we'd like to see if this was possible sooner rather than later - our builds are slow with your current infrastructure, and it seems like a simple change on your side to make them significantly faster.

Hi guys,

It seems you've found a bug with how our new infrastructure presents memory errors, so I'll update this ticket to reflect that. To explain:

• We have always configured the memory limit in Docker using -m as suggested above, which sets the cgroup memory limit and results in the kernel automatically killing any process that exceeds this limit ("OOM killed").
• Any command in Pipelines that is OOM killed would return a failure exit code, which would normally terminate the Pipeline at this point. But in your case, you're using || true (or similar tricks) to continue running the script when this occurs.
• On our previous infrastructure we didn't detect processes were OOM killed as a distinct failure mode, but since we migrated to new infrastructure in the past two weeks, now we do. We check for the "memory exceeded" flag on the container when it completes, and if it does – regardless of the Pipeline exit code – we mark the build as failed based on memory exhaustion.

Our mistaken expectation was that any pipeline that reported an "OOM killed" status would also be a failed build. However, in your case, that isn't true. Your pipeline is ignoring the process failure and continuing to run. So the fix would be for us to switch back to using the pipeline script exit code as the build status, and add the OOM status as a warning on the build instead in the case where the build is successful.

Does that sound correct to you both?

Before I try more debugging can you explain a bit more about why you can't just allocate 8gb - either for us, or for all containers?

Sure, the reason is that we've planned our Pipelines infrastructure and pricing based on allocating 4 GB to each customer. So larger memory allocations is something we could offer only with changes to our planned pricing. Based on the customer feedback we've heard to date, 4 GB seems to be enough for the vast majority of Pipelines users, although we know there are some exceptions.

If you'd like to open a ticket to specifically request increased memory, we can track demand for this and consider it in the future. I'd prefer to keep this ticket tracking the bug with OOM build status reporting, given the relevant technical details included above.

Thanks,
Matt

Hey guys,

Thanks for the responses. In order

Joshua:

• Dependency caching may help although this is internal to our project - so we'd have to split up the repo or similar.
• You're correct on the intermediate commits, this is my mistake

Matt:

• Yes it's sampling at 10Hz, see https://github.com/jhclark/memusg if you're interested (note looking at resident rather than virtual)
• I've spent a bit of time already trying to fit in with the 4gb limit without major changes to our repo or the way we do things here. Before I try more debugging can you explain a bit more about why you can't just allocate 8gb - either for us, or for all containers?

However, we have recently improved the error handling so that memory errors are now reported in the Pipelines UI.

How recently was this made? Last Friday? I think it's what caused my builds to start failing. I had a workaround for OOM in place and working. Now pipelines considers the build failed after oom, even if my script recovers and finishes successfully.

My workaround is to run full-parallel until oom killer stopis it, then try again serially:

#!bash
set -e
make -j4 || true
make

Hi Nick,

I can add two points on the specific memory problem you're having.

First, our handling of builds that exceed the memory limit (failing the build immediately) is unfortunately driven by platform limitations in terms of how Docker works with the Linux kernel. We haven't been able to find a way to provide a softer limit to customers yet. (However, we have recently improved the error handling so that memory errors are now reported in the Pipelines UI.) We'll continue to keep our eyes open for development in this area - please let us know if you have any specific ideas.

Second, your memory monitoring is probably doing sampling and might not get all the allocations that hit the OS. So there may be peak memory usage higher than what your tool is reporting. We'd love to have better monitoring of build resource usage that we could provide to our customers, but unfortunately developing that feature is quite far down our priorities right now. Likewise for offering an interactive shell. We're working on first enabling Docker build and push, and a few other highly voted features.

If you have the time and impetus to investigate this further yourself, the best replica environment is probably spinning up your build container on Amazon ECS, and seeing if you can get the memory metrics you need there to find a way to get your build consistently below 4GB.

If we can help you further at all with this, please let us know.

Regards,
Matt

Hi Nick,

Thanks for the feedback.

There appears to be no nice way to maintain build state, short of seeding the container with object files (eg checking in build artifacts). This means that even a tiny change results in a 40-60 minute build.

We have an open feature request to allow dependencies to be cached between pipelines BCLOUD-12818. Could you let me know if such a solution would help?

Related, if people check in multiple commits we get multiple 40-60 minute builds (is there no build holfoff time as with buildbot? If it doesn't exist already it might be nice to have the option to only build head and not all intermediate commits)

This also sounds like BCLOUD-13463

Now for your OOM error, unfortunately we currently can't increase memory beyond 4g as this means we would need to increase memory for all builds across Pipelines. I'm not familiar enough with -jX and peak memory so I'll have a colleague add a further reply.

Regards,
Joshua

Edit: Actually I just realised we can presumably manage our own cgroups directly in the image, ignore that part