Out of memory errors reported as failed pipelines, even when ignored

Hi guys and @rahulchhabria,

We've got a smallish monolithic repo(~1.5Mloc) that we have been building with pipelines

So far pipelines itself is great and appears to have regular improvements, which we love - we're been tossing up moving away from internal buildbot and relying on this entirely

There are a couple of issues though that we are having trouble with, and have been wondering if there are nice solutions to:

• There appears to be no nice way to maintain build state, short of seeding the container with object files (eg checking in build artifacts). This means that even a tiny change results in a 40-60 minute build.

For this, we understand that letting people have incremental builds leads to a whole host of problems, but it'd be great to have an advanced feature that lets you maintain image state by default (acknowledging that you may get failed builds or weirdness) and only do a full build if specified, e.g. if there is an error

• Related, if people check in multiple commits we get multiple 40-60 minute builds (is there no build holfoff time as with buildbot? If it doesn't exist already it might be nice to have the option to only build head and not all intermediate commits)
• We can't build with any more than -j3 as we get a non-recoverable error halfway through the build that memory is exhausted. Peak build memory usage on our centos6 system with -j6 is 3.6GB (vs 2.6 with -j3 and 4.1 with -j8), so I'm not sure why this doesn't work.

The fact that the error for OOM is fatal, non-recoverable, and takes 20 minutes to occur is itself counter-productive too - a continuable soft limit using cgroups or similar (perhaps including email warnings) would be infinitely preferable

• Having basic interactive console access to the system would be very helpful. Having to edit the pipelines yml, commit, wait for email error, repeat is a slow workflow

Apologies if any of these are covered in documentation - I haven't been through all of it in the last month or so, so will likely have missed things