Uploaded image for project: 'Server Deployments and Scale'
  1. Server Deployments and Scale
  2. SCALE-76

Confluence Pod fails to start when Synchrony is enabled on Helm Chart 1.5.0

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • None
    • master
    • None
    • Severity 2 - Major

      The Confluence Helm Chart fails to start when Synchrony is enabled:

      Helm Chart Value
      synchrony:
        enabled: true
      

      Error

      1. Confluence Pod fails to start:
        % kubectl get all
        NAME                                            READY   STATUS             RESTARTS     AGE
        pod/confluencelocal-0                           0/1     CrashLoopBackOff   3 (6s ago)   60s
        pod/confluencelocal-synchrony-0                 1/1     Running            0            60s
        pod/ingress-nginx-controller-6bf7bc7f94-xbw24   1/1     Running            0            154m
        pod/postgres-postgresql-0                       1/1     Running            0            149m
        ...
        ..
        
      2. Pod describe:
        % kubectl describe pod confluencelocal-0 
        ame:         confluencelocal-0
        Namespace:    confluence
        ...
        ..
        Events:
          Type     Reason     Age                From               Message
          ----     ------     ----               ----               -------
          Normal   Scheduled  77s                default-scheduler  Successfully assigned confluence/confluencelocal-0 to docker-desktop
          Normal   Pulled     76s                kubelet            Container image "alpine" already present on machine
          Normal   Created    76s                kubelet            Created container nfs-permission-fixer
          Normal   Started    76s                kubelet            Started container nfs-permission-fixer
          Normal   Pulled     24s (x4 over 75s)  kubelet            Container image "atlassian/confluence:7.14.1" already present on machine
          Normal   Created    24s (x4 over 75s)  kubelet            Created container confluence
          Normal   Started    24s (x4 over 75s)  kubelet            Started container confluence
          Warning  BackOff    7s (x8 over 71s)   kubelet            Back-off restarting failed container
        
      3. Pod Logs:
        % kubectl logs confluencelocal-0
        Defaulted container "confluence" out of: confluence, nfs-permission-fixer (init)
        INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2
        INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2
        INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2
        INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.
        INFO:root:User is currently root. Will downgrade run user to confluence
        WARNING:root:Unsetting environment var ATL_JDBC_PASSWORD
        INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']
        executing as current user
        If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
        
        Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
        ---------------------------------------------------------------------------
        Using Java: /opt/java/openjdk/bin/java
        2022-07-27 04:03:28,048 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
        ---------------------------------------------------------------------------
        NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
        [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-07-27_04-03-28.log instead.
        Usage: java [options] <mainclass> [args...]
                   (to execute a class)
           or  java [options] -jar <jarfile> [args...]
                   (to execute a jar file)
           or  java [options] -m <module>[/<mainclass>] [args...]
               java [options] --module <module>[/<mainclass>] [args...]
                   (to execute the main class in a module)
           or  java [options] <sourcefile> [args]
                   (to execute a single source-file program)
        
         Arguments following the main class, source file, -jar <jarfile>,
         -m or --module <module>/<mainclass> are passed as the arguments to
         main class.
        
         where options include:
        
            -cp <class search path of directories and zip/jar files>
            -classpath <class search path of directories and zip/jar files>
            --class-path <class search path of directories and zip/jar files>
                          A : separated list of directories, JAR archives,
                          and ZIP archives to search for class files.
            -p <module path>
        ..
        ..
        

      Looks like a java startup error.

      Comparing the config map of a older working Helm Chart and the current Helm Chart:

      Older Working Helm Chart configmap
      % kubectl describe configmap confluencelocal-jvm-config
      ...
      ..
      Data
      ====
      additional_jvm_args:
      ----
      -Dconfluence.cluster.hazelcast.listenPort=5701 -Dsynchrony.service.url=http://localhost/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
      
      Current Helm Chart configmap
      % kubectl describe configmap confluencelocal-jvm-config
      ...
      ..
      Data
      ====
      additional_jvm_args:
      ----
      -Dconfluence.cluster.hazelcast.listenPort=5701
      -Dsynchrony.service.url=http://localhost/synchrony/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
      

      Comparing the changes in src/main/charts/confluence/templates/_helpers.tpl:

      Old working Repo Code
      {{- define "confluence.sysprop.synchronyServiceUrl" -}}
      {{- if .Values.synchrony.enabled -}}
      -Dsynchrony.service.url={{ .Values.synchrony.ingressUrl }}/v1
      {{- else -}}
      -Dsynchrony.btf.disabled=true
      {{- end -}}
      {{- end }}
      
      Latest Repo Code
      {{- define "confluence.sysprop.synchronyServiceUrl" -}}
      {{- if .Values.synchrony.enabled -}}
          {{- if .Values.ingress.https -}}
          -Dsynchrony.service.url=https://{{ .Values.ingress.host }}/synchrony/v1
          {{- else }}
          -Dsynchrony.service.url=http://{{ .Values.ingress.host }}/synchrony/v1
          {{- end }}
      {{- else -}}
      -Dsynchrony.btf.disabled=true
      {{- end -}}
      {{- end -}}
      

      The indentation of the -Dsynchrony.service.url is causing a new line in the <release>-jvm-config which ultimately gets populated into ${JVM_SUPPORT_RECOMMENDED_ARGS} in setenv.sh.

      The code that generates JVM_SUPPORT_RECOMMENDED_ARGS is in statefulset.yaml:

                  - name: JVM_SUPPORT_RECOMMENDED_ARGS
                    valueFrom:
                      configMapKeyRef:
                        key: additional_jvm_args
                        name: {{ include "common.names.fullname" . }}-jvm-config
      

            [SCALE-76] Confluence Pod fails to start when Synchrony is enabled on Helm Chart 1.5.0

            thanks, this was affecting us so we reverted back to 1.4.0 until fixed.

            Steve Letch added a comment - thanks, this was affecting us so we reverted back to 1.4.0 until fixed.

            This has been fixed in 1.5.1 version

            Nasser Ghazali added a comment - This has been fixed in 1.5.1 version

            Nasser Ghazali added a comment - - edited

            Data Center Helm Chart version 1.5.1 for Confluence is released which contains the fix for this problem.

             

            Nasser Ghazali added a comment - - edited Data Center Helm Chart version 1.5.1 for Confluence is released which contains the fix for this problem.  

              Unassigned Unassigned
              hlam@atlassian.com Eric Lam
              Affected customers:
              0 This affects my team
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: