Uploaded image for project: 'Server Deployments and Scale'
  1. Server Deployments and Scale
  2. SCALE-76

Confluence Pod fails to start when Synchrony is enabled on Helm Chart 1.5.0

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • None
    • master
    • None
    • Severity 2 - Major

      The Confluence Helm Chart fails to start when Synchrony is enabled:

      Helm Chart Value
      synchrony:
        enabled: true
      

      Error

      1. Confluence Pod fails to start:
        % kubectl get all
        NAME                                            READY   STATUS             RESTARTS     AGE
        pod/confluencelocal-0                           0/1     CrashLoopBackOff   3 (6s ago)   60s
        pod/confluencelocal-synchrony-0                 1/1     Running            0            60s
        pod/ingress-nginx-controller-6bf7bc7f94-xbw24   1/1     Running            0            154m
        pod/postgres-postgresql-0                       1/1     Running            0            149m
        ...
        ..
        
      2. Pod describe:
        % kubectl describe pod confluencelocal-0 
        ame:         confluencelocal-0
        Namespace:    confluence
        ...
        ..
        Events:
          Type     Reason     Age                From               Message
          ----     ------     ----               ----               -------
          Normal   Scheduled  77s                default-scheduler  Successfully assigned confluence/confluencelocal-0 to docker-desktop
          Normal   Pulled     76s                kubelet            Container image "alpine" already present on machine
          Normal   Created    76s                kubelet            Created container nfs-permission-fixer
          Normal   Started    76s                kubelet            Started container nfs-permission-fixer
          Normal   Pulled     24s (x4 over 75s)  kubelet            Container image "atlassian/confluence:7.14.1" already present on machine
          Normal   Created    24s (x4 over 75s)  kubelet            Created container confluence
          Normal   Started    24s (x4 over 75s)  kubelet            Started container confluence
          Warning  BackOff    7s (x8 over 71s)   kubelet            Back-off restarting failed container
        
      3. Pod Logs:
        % kubectl logs confluencelocal-0
        Defaulted container "confluence" out of: confluence, nfs-permission-fixer (init)
        INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2
        INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2
        INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2
        INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.
        INFO:root:User is currently root. Will downgrade run user to confluence
        WARNING:root:Unsetting environment var ATL_JDBC_PASSWORD
        INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']
        executing as current user
        If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
        
        Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
        ---------------------------------------------------------------------------
        Using Java: /opt/java/openjdk/bin/java
        2022-07-27 04:03:28,048 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
        ---------------------------------------------------------------------------
        NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
        [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-07-27_04-03-28.log instead.
        Usage: java [options] <mainclass> [args...]
                   (to execute a class)
           or  java [options] -jar <jarfile> [args...]
                   (to execute a jar file)
           or  java [options] -m <module>[/<mainclass>] [args...]
               java [options] --module <module>[/<mainclass>] [args...]
                   (to execute the main class in a module)
           or  java [options] <sourcefile> [args]
                   (to execute a single source-file program)
        
         Arguments following the main class, source file, -jar <jarfile>,
         -m or --module <module>/<mainclass> are passed as the arguments to
         main class.
        
         where options include:
        
            -cp <class search path of directories and zip/jar files>
            -classpath <class search path of directories and zip/jar files>
            --class-path <class search path of directories and zip/jar files>
                          A : separated list of directories, JAR archives,
                          and ZIP archives to search for class files.
            -p <module path>
        ..
        ..
        

      Looks like a java startup error.

      Comparing the config map of a older working Helm Chart and the current Helm Chart:

      Older Working Helm Chart configmap
      % kubectl describe configmap confluencelocal-jvm-config
      ...
      ..
      Data
      ====
      additional_jvm_args:
      ----
      -Dconfluence.cluster.hazelcast.listenPort=5701 -Dsynchrony.service.url=http://localhost/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
      
      Current Helm Chart configmap
      % kubectl describe configmap confluencelocal-jvm-config
      ...
      ..
      Data
      ====
      additional_jvm_args:
      ----
      -Dconfluence.cluster.hazelcast.listenPort=5701
      -Dsynchrony.service.url=http://localhost/synchrony/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
      

      Comparing the changes in src/main/charts/confluence/templates/_helpers.tpl:

      Old working Repo Code
      {{- define "confluence.sysprop.synchronyServiceUrl" -}}
      {{- if .Values.synchrony.enabled -}}
      -Dsynchrony.service.url={{ .Values.synchrony.ingressUrl }}/v1
      {{- else -}}
      -Dsynchrony.btf.disabled=true
      {{- end -}}
      {{- end }}
      
      Latest Repo Code
      {{- define "confluence.sysprop.synchronyServiceUrl" -}}
      {{- if .Values.synchrony.enabled -}}
          {{- if .Values.ingress.https -}}
          -Dsynchrony.service.url=https://{{ .Values.ingress.host }}/synchrony/v1
          {{- else }}
          -Dsynchrony.service.url=http://{{ .Values.ingress.host }}/synchrony/v1
          {{- end }}
      {{- else -}}
      -Dsynchrony.btf.disabled=true
      {{- end -}}
      {{- end -}}
      

      The indentation of the -Dsynchrony.service.url is causing a new line in the <release>-jvm-config which ultimately gets populated into ${JVM_SUPPORT_RECOMMENDED_ARGS} in setenv.sh.

      The code that generates JVM_SUPPORT_RECOMMENDED_ARGS is in statefulset.yaml:

                  - name: JVM_SUPPORT_RECOMMENDED_ARGS
                    valueFrom:
                      configMapKeyRef:
                        key: additional_jvm_args
                        name: {{ include "common.names.fullname" . }}-jvm-config
      

            [SCALE-76] Confluence Pod fails to start when Synchrony is enabled on Helm Chart 1.5.0

            thanks, this was affecting us so we reverted back to 1.4.0 until fixed.

            Steve Letch added a comment - thanks, this was affecting us so we reverted back to 1.4.0 until fixed.
            Nasser Ghazali made changes -
            Resolution New: Fixed [ 1 ]
            Status Original: Needs Triage [ 10030 ] New: Closed [ 6 ]

            This has been fixed in 1.5.1 version

            Nasser Ghazali added a comment - This has been fixed in 1.5.1 version

            Nasser Ghazali added a comment - - edited

            Data Center Helm Chart version 1.5.1 for Confluence is released which contains the fix for this problem.

             

            Nasser Ghazali added a comment - - edited Data Center Helm Chart version 1.5.1 for Confluence is released which contains the fix for this problem.  
            Steve Smith (Inactive) made changes -
            Remote Link New: This issue links to "CLIP-1632 (JIRA Server (Bulldog))" [ 670338 ]
            Eric Lam made changes -
            Description Original: The Confluence Helm Chart fails to start when Synchrony is enabled:

            {code:title=Helm Chart Value}
            synchrony:
              enabled: true
            {code}

            h3. Error

            # Confluence Pod fails to start:
            {code}
            % kubectl get all
            NAME READY STATUS RESTARTS AGE
            pod/confluencelocal-0 0/1 CrashLoopBackOff 3 (6s ago) 60s
            pod/confluencelocal-synchrony-0 1/1 Running 0 60s
            pod/ingress-nginx-controller-6bf7bc7f94-xbw24 1/1 Running 0 154m
            pod/postgres-postgresql-0 1/1 Running 0 149m
            ...
            ..
            {code}
            # Pod describe:
            {code}
            % kubectl describe pod confluencelocal-0
            ame: confluencelocal-0
            Namespace: confluence
            ...
            ..
            Events:
              Type Reason Age From Message
              ---- ------ ---- ---- -------
              Normal Scheduled 77s default-scheduler Successfully assigned confluence/confluencelocal-0 to docker-desktop
              Normal Pulled 76s kubelet Container image "alpine" already present on machine
              Normal Created 76s kubelet Created container nfs-permission-fixer
              Normal Started 76s kubelet Started container nfs-permission-fixer
              Normal Pulled 24s (x4 over 75s) kubelet Container image "atlassian/confluence:7.14.1" already present on machine
              Normal Created 24s (x4 over 75s) kubelet Created container confluence
              Normal Started 24s (x4 over 75s) kubelet Started container confluence
              Warning BackOff 7s (x8 over 71s) kubelet Back-off restarting failed container
            {code}
            # Pod Logs:
            {code}
            % kubectl logs confluencelocal-0
            Defaulted container "confluence" out of: confluence, nfs-permission-fixer (init)
            INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2
            INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2
            INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2
            INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.
            INFO:root:User is currently root. Will downgrade run user to confluence
            WARNING:root:Unsetting environment var ATL_JDBC_PASSWORD
            INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']
            executing as current user
            If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

            Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
            ---------------------------------------------------------------------------
            Using Java: /opt/java/openjdk/bin/java
            2022-07-27 04:03:28,048 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
            ---------------------------------------------------------------------------
            NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
            [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-07-27_04-03-28.log instead.
            Usage: java [options] <mainclass> [args...]
                       (to execute a class)
               or java [options] -jar <jarfile> [args...]
                       (to execute a jar file)
               or java [options] -m <module>[/<mainclass>] [args...]
                   java [options] --module <module>[/<mainclass>] [args...]
                       (to execute the main class in a module)
               or java [options] <sourcefile> [args]
                       (to execute a single source-file program)

             Arguments following the main class, source file, -jar <jarfile>,
             -m or --module <module>/<mainclass> are passed as the arguments to
             main class.

             where options include:

                -cp <class search path of directories and zip/jar files>
                -classpath <class search path of directories and zip/jar files>
                --class-path <class search path of directories and zip/jar files>
                              A : separated list of directories, JAR archives,
                              and ZIP archives to search for class files.
                -p <module path>
            ..
            ..
            {code}

            Looks like a java startup error.

            Comparing the config map of a older working Helm Chart and the current Helm Chart:

            {code:title=Older Working Helm Chart configmap (/)}
            % kubectl describe configmap confluencelocal-jvm-config
            ...
            ..
            Data
            ====
            additional_jvm_args:
            ----
            -Dconfluence.cluster.hazelcast.listenPort=5701 -Dsynchrony.service.url=http://localhost/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
            {code}

            {code:title=Current Helm Chart configmap (x)}
            % kubectl describe configmap confluencelocal-jvm-config
            ...
            ..
            Data
            ====
            additional_jvm_args:
            ----
            -Dconfluence.cluster.hazelcast.listenPort=5701
            -Dsynchrony.service.url=http://localhost/synchrony/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
            {code}

            Comparing the changes in *{{src/main/charts/confluence/templates/_helpers.tpl}}*:

            {code:title=Old working Repo Code}
            {{- define "confluence.sysprop.synchronyServiceUrl" -}}
            {{- if .Values.synchrony.enabled -}}
            -Dsynchrony.service.url={{ .Values.synchrony.ingressUrl }}/v1
            {{- else -}}
            -Dsynchrony.btf.disabled=true
            {{- end -}}
            {{- end }}
            {code}

            {code:title=Latest Repo Code}
            {{- define "confluence.sysprop.synchronyServiceUrl" -}}
            {{- if .Values.synchrony.enabled -}}
                {{- if .Values.ingress.https -}}
                -Dsynchrony.service.url=https://{{ .Values.ingress.host }}/synchrony/v1
                {{- else }}
                -Dsynchrony.service.url=http://{{ .Values.ingress.host }}/synchrony/v1
                {{- end }}
            {{- else -}}
            -Dsynchrony.btf.disabled=true
            {{- end -}}
            {{- end -}}
            {code}

            The indentation of the {{-Dsynchrony.service.url}} is causing a new line in the {{<release>--jvm-config}} which ultimately gets populated into {{$\{JVM_SUPPORT_RECOMMENDED_ARGS\}}} in *setenv.sh*.
            New: The Confluence Helm Chart fails to start when Synchrony is enabled:

            {code:title=Helm Chart Value}
            synchrony:
              enabled: true
            {code}

            h3. Error

            # Confluence Pod fails to start:
            {code}
            % kubectl get all
            NAME READY STATUS RESTARTS AGE
            pod/confluencelocal-0 0/1 CrashLoopBackOff 3 (6s ago) 60s
            pod/confluencelocal-synchrony-0 1/1 Running 0 60s
            pod/ingress-nginx-controller-6bf7bc7f94-xbw24 1/1 Running 0 154m
            pod/postgres-postgresql-0 1/1 Running 0 149m
            ...
            ..
            {code}
            # Pod describe:
            {code}
            % kubectl describe pod confluencelocal-0
            ame: confluencelocal-0
            Namespace: confluence
            ...
            ..
            Events:
              Type Reason Age From Message
              ---- ------ ---- ---- -------
              Normal Scheduled 77s default-scheduler Successfully assigned confluence/confluencelocal-0 to docker-desktop
              Normal Pulled 76s kubelet Container image "alpine" already present on machine
              Normal Created 76s kubelet Created container nfs-permission-fixer
              Normal Started 76s kubelet Started container nfs-permission-fixer
              Normal Pulled 24s (x4 over 75s) kubelet Container image "atlassian/confluence:7.14.1" already present on machine
              Normal Created 24s (x4 over 75s) kubelet Created container confluence
              Normal Started 24s (x4 over 75s) kubelet Started container confluence
              Warning BackOff 7s (x8 over 71s) kubelet Back-off restarting failed container
            {code}
            # Pod Logs:
            {code}
            % kubectl logs confluencelocal-0
            Defaulted container "confluence" out of: confluence, nfs-permission-fixer (init)
            INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2
            INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2
            INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2
            INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.
            INFO:root:User is currently root. Will downgrade run user to confluence
            WARNING:root:Unsetting environment var ATL_JDBC_PASSWORD
            INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']
            executing as current user
            If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

            Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
            ---------------------------------------------------------------------------
            Using Java: /opt/java/openjdk/bin/java
            2022-07-27 04:03:28,048 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
            ---------------------------------------------------------------------------
            NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
            [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-07-27_04-03-28.log instead.
            Usage: java [options] <mainclass> [args...]
                       (to execute a class)
               or java [options] -jar <jarfile> [args...]
                       (to execute a jar file)
               or java [options] -m <module>[/<mainclass>] [args...]
                   java [options] --module <module>[/<mainclass>] [args...]
                       (to execute the main class in a module)
               or java [options] <sourcefile> [args]
                       (to execute a single source-file program)

             Arguments following the main class, source file, -jar <jarfile>,
             -m or --module <module>/<mainclass> are passed as the arguments to
             main class.

             where options include:

                -cp <class search path of directories and zip/jar files>
                -classpath <class search path of directories and zip/jar files>
                --class-path <class search path of directories and zip/jar files>
                              A : separated list of directories, JAR archives,
                              and ZIP archives to search for class files.
                -p <module path>
            ..
            ..
            {code}

            Looks like a java startup error.

            Comparing the config map of a older working Helm Chart and the current Helm Chart:

            {code:title=Older Working Helm Chart configmap (/)}
            % kubectl describe configmap confluencelocal-jvm-config
            ...
            ..
            Data
            ====
            additional_jvm_args:
            ----
            -Dconfluence.cluster.hazelcast.listenPort=5701 -Dsynchrony.service.url=http://localhost/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
            {code}

            {code:title=Current Helm Chart configmap (x)}
            % kubectl describe configmap confluencelocal-jvm-config
            ...
            ..
            Data
            ====
            additional_jvm_args:
            ----
            -Dconfluence.cluster.hazelcast.listenPort=5701
            -Dsynchrony.service.url=http://localhost/synchrony/v1 -Dsynchrony.by.default.enable.collab.editing.if.manually.managed=true -Dconfluence.clusterNodeName.useHostname=true -Datlassian.logging.cloud.enabled=false -XX:ActiveProcessorCount=1
            {code}

            Comparing the changes in *{{src/main/charts/confluence/templates/_helpers.tpl}}*:

            {code:title=Old working Repo Code}
            {{- define "confluence.sysprop.synchronyServiceUrl" -}}
            {{- if .Values.synchrony.enabled -}}
            -Dsynchrony.service.url={{ .Values.synchrony.ingressUrl }}/v1
            {{- else -}}
            -Dsynchrony.btf.disabled=true
            {{- end -}}
            {{- end }}
            {code}

            {code:title=Latest Repo Code}
            {{- define "confluence.sysprop.synchronyServiceUrl" -}}
            {{- if .Values.synchrony.enabled -}}
                {{- if .Values.ingress.https -}}
                -Dsynchrony.service.url=https://{{ .Values.ingress.host }}/synchrony/v1
                {{- else }}
                -Dsynchrony.service.url=http://{{ .Values.ingress.host }}/synchrony/v1
                {{- end }}
            {{- else -}}
            -Dsynchrony.btf.disabled=true
            {{- end -}}
            {{- end -}}
            {code}

            The indentation of the {{\-Dsynchrony.service.url}} is causing a new line in the {{<release>\-jvm\-config}} which ultimately gets populated into {{$\{JVM_SUPPORT_RECOMMENDED_ARGS\}}} in *setenv.sh*.

            The code that generates JVM_SUPPORT_RECOMMENDED_ARGS is in statefulset.yaml:

            {code}
                        - name: JVM_SUPPORT_RECOMMENDED_ARGS
                          valueFrom:
                            configMapKeyRef:
                              key: additional_jvm_args
                              name: {{ include "common.names.fullname" . }}-jvm-config
            {code}
            Eric Lam created issue -

              Unassigned Unassigned
              hlam@atlassian.com Eric Lam
              Affected customers:
              0 This affects my team
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: