Problem

Bitbucket nodes under heavy load can be stopped due to running out of system memory. This is caused by forked git operations consuming all available resources.

There are various ways Git operations can consume a lot of memory. These usually involve repos that are huge in size (10GB or more). git repack and git pack-objects are operations that may potentially consume 10GB, or in extreme cases 40GB of RAM.

Suggested Solution

HTTP/SSH requests which forked a git process consuming a huge amount of memory should be logged with a warning message. The log entry should mention the repository involved so an Admin can investigate further. 

Why this is important

These log entries help in early detection of issues caused by large repositories. By being aware of repositories consuming a lot of RAM, an Admin can take necessary action such as:

• Increasing the RAM
• Adjusting the hosting tickets allocated per node
• Reducing the frequency or adjusting the schedule by which CI/CD pulls data out of Bitbucket
• Adding more nodes or setting up mirrors

Workaround

• Use ps -ef to monitor running processes
• Capture the PID of the git process consuming a huge amount of RAM
• Run pwdx <pid> to get the working directory of the process
• cat repository-config to get the name of the repository