Bamboo start-up scripts misuse JAVA_OPTS which can cause Out of Memory when stopping Bamboo

Issue Summary

Bamboo start-up scripts (setenv.sh) misuses JAVA_OPTS for all JVM arguments. Forked Java processes (e.g. Gradle) on local agents will inherit Bamboo JVM heap and proxy settings

It also has the side-effect of causing the Tomcat stop command to require the same initial heap as the Bamboo JVM. This will cause OOMs (mostly where Xms=Xmx) and not a sufficient amount of available memory to create a second JVM with the same memory requirements.

Expected Results

Use CATALINA_OPTS instead so that the Tomcat stop command does not require the same amount of memory as configured for Bamboo.

Actual Results

Tomcat stop ($BAMBOO_INSTALL/bin/stop-bamboo.sh / shutdown.sh) may Out Of Memory when initializing the heap:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1431306240 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=4426, tid=0x00007f42b0539700
#
# JRE version:  (8.0_162-b12) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x00007f42a800a000):  JavaThread "Unknown thread" [_thread_in_vm, id=4427, stack(0x00007f42b043a000,0x00007f42b053a000)]

Stack: [0x00007f42b043a000,0x00007f42b053a000],  sp=0x00007f42b05383c0,  free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xacfb2a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x50060b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V  [libjvm.so+0x92d963]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x123
V  [libjvm.so+0x92db89]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V  [libjvm.so+0x924e4a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V  [libjvm.so+0x99adf3]  PSVirtualSpace::expand_by(unsigned long)+0x53
V  [libjvm.so+0x99bdb5]  PSYoungGen::initialize_virtual_space(ReservedSpace, unsigned long)+0x75
V  [libjvm.so+0x99c71e]  PSYoungGen::initialize(ReservedSpace, unsigned long)+0x3e
V  [libjvm.so+0x2db2b5]  AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, GenerationSizer*, unsigned long)+0x345
V  [libjvm.so+0x94ef36]  ParallelScavengeHeap::initialize()+0x1d6
V  [libjvm.so+0xa97bd3]  Universe::initialize_heap()+0xf3
V  [libjvm.so+0xa9813e]  universe_init()+0x3e
V  [libjvm.so+0x6420f5]  init_globals()+0x65
V  [libjvm.so+0xa7c8ee]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
V  [libjvm.so+0x6d6cc4]  JNI_CreateJavaVM+0x74
C  [libjli.so+0x797e]  JavaMain+0x9e
C  [libpthread.so.0+0x7de5]  start_thread+0xc5

Workaround

Option 1

Set JVM_MINIMUM_MEMORY in $BAMBOO_INSTALL/bin/setenv.sh to a smaller value (e.g. 512m) so that the Tomcat stop command requires less memory but Bamboo can still scale up to the value of JVM_MAXIMUM_MEMORY.

Option 2

Manually patch this part of your $BAMBOO_INSTALL/bin/setenv.sh to not use JAVA_OPTS for memory settings.

• Backup your setenv.sh before making any modifications

JAVA_OPTS="-Xms${JVM_MINIMUM_MEMORY} -Xmx${JVM_MAXIMUM_MEMORY} ${JAVA_OPTS} ${JVM_REQUIRED_ARGS} ${JVM_SUPPORT_RECOMMENDED_ARGS} ${BAMBOO_HOME_MINUSD}"

JAVA_OPTS=$(echo "$JAVA_OPTS" | sed -e 's/\s*$//' -e 's/^\s*//')
export JAVA_OPTS

JAVA_OPTS="${JAVA_OPTS} ${JVM_REQUIRED_ARGS} ${JVM_SUPPORT_RECOMMENDED_ARGS} ${BAMBOO_HOME_MINUSD}"
JAVA_OPTS=$(echo "$JAVA_OPTS" | sed -e 's/\s*$//' -e 's/^\s*//')

CATALINA_OPTS="-Xms${JVM_MINIMUM_MEMORY} -Xmx${JVM_MAXIMUM_MEMORY} ${CATALINA_OPTS}"
CATALINA_OPTS=$(echo "$CATALINA_OPTS" | sed -e 's/\s*$//' -e 's/^\s*//')

export JAVA_OPTS
export CATALINA_OPTS

• This example only migrates the Memory settings to CATALINA_OPTS, you may choose to move JVM_SUPPORT_RECOMMENDED_ARGS to CATALINA_OPTS too if you prefer those arguments to not be inherited by other Java processes forked from Bamboo.

Option 3

Run a modified shutdown command.

1. Run bash -x ./catalina.sh stop
2. Make sure Bamboo does not stop at the end.
3. Look for the java command that is being run after an eval command that end with stop, here is an example:

   + '[' '!' -z '' ']'
   + eval '"/usr/lib/jvm/java-8-oracle/bin/java"' -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms9512m -Xmx91024m -Djdk.tls.ephemeralDHKeySize=2048 '-Djava.endorsed.dirs="/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/endorsed"' -classpath '"/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/bin/bootstrap.jar:/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/bin/tomcat-juli.jar"' '-Dcatalina.base="/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3"' '-Dcatalina.home="/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3"' '-Djava.io.tmpdir="/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/temp"' org.apache.catalina.startup.Bootstrap stop
   ++ /usr/lib/jvm/java-8-oracle/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms9512m -Xmx91024m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.endorsed.dirs=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/endorsed -classpath /home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/bin/bootstrap.jar:/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/bin/tomcat-juli.jar -Dcatalina.base=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3 -Dcatalina.home=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3 -Djava.io.tmpdir=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/temp org.apache.catalina.startup.Bootstrap stop
   

4. Take the full Java Command and remove the -Xms only from this command.
5. Run the command. Here is the example:

   /usr/lib/jvm/java-8-oracle/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx91024m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.endorsed.dirs=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/endorsed -classpath /home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/bin/bootstrap.jar:/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/bin/tomcat-juli.jar -Dcatalina.base=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3 -Dcatalina.home=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3 -Djava.io.tmpdir=/home/ubuntu/atlassian/bamboo/atlassian-bamboo-6.6.3/temp org.apache.catalina.startup.Bootstrap stop