We are getting regular OutOfMemory errors that are killing a recently installed Stash. There are only two developers using Stash very lightly so it's not under any load. Using VisualVM I can see the saw-tooth heap/GC getting narrower and narrower until it runs out of room.

I have generated the Heap dump as instructed by https://confluence.atlassian.com/display/STASHKB/How+to+debug+Out+of+Memory+Heap+Space?utm_source=SAC&utm_medium=dashboard

It is uploaded to Google Drive at: https://drive.google.com/file/d/0B2xbHWqkiRFoZ3hULUN4WUhGLUk/edit?usp=sharing

JVM_MINIMUM_MEMORY (Xms) is currently 512m
JVM_MAXIMUM_MEMORY (Xmx) is currently 1536m
STASH_MAX_PERM_SIZE is currently 256m

I am seeing this error in the atlassian-stash.log:
2014-07-02 09:04:19,551 ERROR [threadpool:thread-21118] c.a.s.i.c.StateTransferringExecutor Error while processing asynchronous task
java.lang.OutOfMemoryError: Java heap space

And in the catalina.out I see line upon line of OutofMemory exceptions eg:
java.lang.OutOfMemoryError: Java heap space
3319 Dumping heap to /tmp/stash-heap-dump ...
3320 Heap dump file created [1632662575 bytes in 50.977 secs]
3321 #
3322 # java.lang.OutOfMemoryError: Java heap space
3323 # -XX:OnOutOfMemoryError="kill -3 %p"
3324 # Executing /bin/sh -c "kill -3 25154"...
3325 2014-06-29 12:19:40
3326 Full thread dump OpenJDK 64-Bit Server VM (24.45-b08 mixed mode):

…..

5398 "VM Thread" prio=10 tid=0x00007fe674071000 nid=0x6246 runnable
5399
5400 "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fe674020000 nid=0x6244 runnable
5401
5402 "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fe674022000 nid=0x6245 runnable
5403
5404 "VM Periodic Task Thread" prio=10 tid=0x00007fe674185000 nid=0x6250 waiting on condition
5405
5406 JNI global references: 370
5407
5408 Heap
5409 PSYoungGen total 406528K, used 398334K [0x00000000e0000000, 0x00000000fa180000, 0x0000000100000000)
5410 eden space 398336K, 99% used [0x00000000e0000000,0x00000000f84ffae0,0x00000000f8500000)
5411 from space 8192K, 0% used [0x00000000f9180000,0x00000000f9180000,0x00000000f9980000)
5412 to space 8192K, 0% used [0x00000000f9980000,0x00000000f9980000,0x00000000fa180000)
5413 ParOldGen total 1048576K, used 1048565K [0x00000000a0000000, 0x00000000e0000000, 0x00000000e0000000)
5414 object space 1048576K, 99% used [0x00000000a0000000,0x00000000dfffd748,0x00000000e0000000)
5415 PSPermGen total 141312K, used 141017K [0x0000000090000000, 0x0000000098a00000, 0x00000000a0000000)
5416 object space 141312K, 99% used [0x0000000090000000,0x00000000989b6470,0x0000000098a00000)
5417
5418 Jun 29, 2014 12:21:30 PM ServerCommunicatorAdmin reqIncoming
5419 WARNING: The server has decided to close this client connection.
5420 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
5421 Exception in thread "httpclient-io:thread-2" java.lang.OutOfMemoryError: Java heap space
5422 Jun 30, 2014 7:25:38 AM ServerCommunicatorAdmin reqIncoming
5423 WARNING: The server has decided to close this client connection.
5424 Exception in thread "scheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: Java heap space
5425 Exception in thread "httpclient-io:thread-4" Exception in thread "httpclient-io:thread-9" java.lang.OutOfMemoryError: Java heap space
5426 java.lang.OutOfMemoryError: Java heap space
5427 Jun 30, 2014 7:46:58 PM ServerCommunicatorAdmin reqIncoming
5428 WARNING: The server has decided to close this client connection.
5429 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
5430 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
5431 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
5432 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
5433 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
5434 Exception in thread "RMI TCP Connection(idle)"
5435 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "httpclient-io:thread-5"
5436
5437 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
5438 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
5439 Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space