Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-13623

Provide better way to monitor JMX metrics for Bitbucket 8.x while using Mesh Sidecar

    • Icon: Suggestion Suggestion
    • Resolution: Unresolved
    • None
    • Administration, Mesh
    • None
    • 70
    • We collect Bitbucket feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      As of Bitbucket 8.0, Git operations are now run in the mesh sidecar by default. This sidecar JVM is launched directly from the Bitbucket JVM, and therefore does not have startup scripts. This means users have to configure the sidecar JVM options through the Bitbucket server(bitbucket.properties) for items such as JMX settings, heap dump, or gc logging configuration.

      To enable JMX monitoring for Bitbucket 8.x, JMX must be enabled on both the Bitbucket main process and the Mesh sidecar process.
      However, adding JMX-related parameters for the Mesh Sidecar can only be done via the "bitbucket.properties" file. This can be a bit complicated if you have a multi-node setup, as the "bitbucket.properties" file is shared and only allows you to define one IP address.

      Option 1:

      To overcome this challenge, each Bitbucket Node must have a local "bitbucket.properties" file with the corresponding local Node IP and then be symbolically linked to the BITBUCKET_HOME/shared/ directory.

      Here are the steps to enable JMX monitoring on the Bitbucket Mesh Sidecar:

      • Enable JMX on the "BITBUCKET_HOME/mesh/mesh.properties" files on all the Nodes by adding the following line:
        jmx.enabled=true
      • Modify the "bitbucket.properties" file as follows to add JVM arguments to the Mesh Sidecar Java process:
        plugin.bitbucket-git.mesh.sidecar.jvmArgs[0]=-Dcom.sun.management.jmxremote.port=4444
        plugin.bitbucket-git.mesh.sidecar.jvmArgs[1]=-Dcom.sun.management.jmxremote.rmi.port=4444
        plugin.bitbucket-git.mesh.sidecar.jvmArgs[2]=-Djava.rmi.server.hostname=<node_ip_address>
        plugin.bitbucket-git.mesh.sidecar.jvmArgs[3]=-Dcom.sun.management.jmxremote.ssl=false
        plugin.bitbucket-git.mesh.sidecar.jvmArgs[4]=-Dcom.sun.management.jmxremote.password.file=/var/atlassian/bitbucket_jmx.access
        

      Option 2:

      Alternatively, you can include the JMX-related configuration in a local bitbucket.properties file within the $BITBUCKET_HOME directory itself.

      To do this, create a $BITBUCKET_HOME/bitbucket.properties file and move all the JMX-related parameters mentioned above to this file. At the same time, in the $BITBUCKET_HOME/shared/bitbucket.properties file, only the common parameters and configurations that are consistent across all other Bitbucket nodes should remain.

      For more information on adding JVM arguments to the Mesh sidecar process, please refer to the article How to pass JVM arguments to the Mesh sidecar.

      After restarting, you should now see JMX enabled on the Bitbucket Mesh sidecar process.

      The request is to make JMX monitoring on Bitbucket 8.x with the Mesh sidecar much cleaner or easier without having to maintain a separate "bitbucket.properties" file for each node.

            [BSERV-13623] Provide better way to monitor JMX metrics for Bitbucket 8.x while using Mesh Sidecar

            even with the new structure, the mbean values for git-lfs tickets ( total, available, used, queued) will still not updated.
            That means , a monitoring of the git-lfs ticket pool is NOT possible at all.

            Robby Berner added a comment - even with the new structure, the mbean values for git-lfs tickets ( total, available, used, queued) will still not updated. That means , a monitoring of the git-lfs ticket pool is NOT possible at all.

            Robby Berner added a comment - - edited

            Can you please better describe, what you mean with:

            To overcome this challenge, each Bitbucket Node must have a local "bitbucket.properties" file with the corresponding local Node IP and then be symbolically linked to the BITBUCKET_HOME/shared/ directory

            So I think, we would need some kind of overlay between some local settings ( the arguments for local mesh sidecar) and the general setting from $BITBUCKET_HOME/shared. How do you want to achieve this?

            Wouldn't it better to start he sidecar also with a start script like bitbucket where you can pass all the JVM arguments easily ? I am talking about set-jmx-opts.sh

            Robby Berner added a comment - - edited Can you please better describe, what you mean with: To overcome this challenge, each Bitbucket Node must have a local "bitbucket.properties" file with the corresponding local Node IP and then be symbolically linked to the BITBUCKET_HOME/shared/ directory So I think, we would need some kind of overlay between some local settings ( the arguments for local mesh sidecar) and the general setting from $BITBUCKET_HOME/shared. How do you want to achieve this? Wouldn't it better to start he sidecar also with a start script like bitbucket where you can pass all the JVM arguments easily ? I am talking about set-jmx-opts.sh

              Unassigned Unassigned
              5338c0aa34f9 Danny Samuel
              Votes:
              21 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated: