Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-13311

Amazon Linux Log4j Hotpatch causes warning logging from the bundled search server

    XMLWordPrintable

Details

    Description

      Issue Summary

      The Amazon Linux Log4j hotpatch causes the bundled search server to log warnings on startup. These warnings are due to the hotpatch being unable to connect to the bundled search server.

      This is reproducible on Data Center: yes (Single Node Data Center with bundled search server)

      Steps to Reproduce

      Start Bitbucket Server (with the bundled search server) on Amazon Linux.

      Expected Results

      Bitbucket and the bundled search server start without any errors or warnings.

      Actual Results

      A few seconds after the Bitbucket startup script completes, the following is logged to the console:

      Agent failed to start!
      

      and the following is logged in the bundled search server logs (at BITBUCKET_HOME/log/search/bitbucket_search.log):

      [2022-05-11T19:59:08,177][WARN ][stderr] [bitbucket_bundled] java.lang.reflect.InvocationTargetException
      [2022-05-11T19:59:08,178][WARN ][stderr] [bitbucket_bundled] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [2022-05-11T19:59:08,179][WARN ][stderr] [bitbucket_bundled] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      [2022-05-11T19:59:08,180][WARN ][stderr] [bitbucket_bundled] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      [2022-05-11T19:59:08,181][WARN ][stderr] [bitbucket_bundled] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      [2022-05-11T19:59:08,183][WARN ][stderr] [bitbucket_bundled] 	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
      [2022-05-11T19:59:08,184][WARN ][stderr] [bitbucket_bundled] 	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
      [2022-05-11T19:59:08,194][WARN ][stderr] [bitbucket_bundled] Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.org.objectweb.asm")
      [2022-05-11T19:59:08,198][WARN ][stderr] [bitbucket_bundled] 	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
      [2022-05-11T19:59:08,199][WARN ][stderr] [bitbucket_bundled] 	at java.base/java.security.AccessController.checkPermission(AccessController.java:897)
      [2022-05-11T19:59:08,199][WARN ][stderr] [bitbucket_bundled] 	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
      [2022-05-11T19:59:08,199][WARN ][stderr] [bitbucket_bundled] 	at java.base/java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1238)
      [2022-05-11T19:59:08,200][WARN ][stderr] [bitbucket_bundled] 	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:174)
      [2022-05-11T19:59:08,200][WARN ][stderr] [bitbucket_bundled] 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
      [2022-05-11T19:59:08,206][WARN ][stderr] [bitbucket_bundled] 	at Log4jHotPatch.asmVersion(Log4jHotPatch.java:71)
      [2022-05-11T19:59:08,207][WARN ][stderr] [bitbucket_bundled] 	at Log4jHotPatch.agentmain(Log4jHotPatch.java:93)
      [2022-05-11T19:59:08,207][WARN ][stderr] [bitbucket_bundled] 	... 6 more
      

      Note that search functionality is not impacted.

      Workaround

      The warnings are indicating that the hotpatch is unable to work with the bundled search server. However, the bundled search server that ships with Bitbucket already has mitigations in place for Bitbucket versions that have addressed the log4j CVE that impacted Elasticsearch and OpenSearch. Running a version of Bitbucket (at least a patch version listed in the fixed versions on the ticket, or Bitbucket 7.20 or higher) that includes these mitigations mean the warnings can be ignored.

      Additionally, it is possible to disable the hotpatch, which in turn will remove the warnings. If Bitbucket and the bundled search server are the only applications running on the machine this is an option. If other applicaitons are running on the same machine (which in general is not recommended) then proper assessment of the other applications should be undertaken before considering disabling the hotpatch.

      It is also possible to use a different operating system as this hotpatch is only part of Amazon Linux.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              de843d56add4 Kalyan Kumar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: