Uploaded image for project: 'Crowd Data Center'
  1. Crowd Data Center
  2. CWD-3983

XML deserialisation happens in the Crowd client in a non-optimal way, resulting in WebappClassLoader lock congestion

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • 2.8
    • None
    • None
    • None

      Problem

      If the cookie.tokenkey is not set in crowd.properties, any Atlassian application integrating with it will perform additional calls to the remote Crowd instance to get it from the cookie configuration. In this scenario it's possible an XML deserialisation will happen in the Crowd client in a non-optimal way, resulting in WebappClassLoader lock congestion. This can cause an instance to become unresponsive ("crash").

      Verification

      Taking thread dumps during a problematic period and analysing them will show the below stack traces:

      "catalina-exec-7615" daemon prio=10 tid=0x00002aaae5ba2000 nid=0xa4c waiting for monitor entry [0x0000000066d72000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java)
	- waiting to lock <0x00000005453089d8> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1556)
	at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
	at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
	at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
	at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
	at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
	at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.<init>(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source)
	at javax.xml.bind.helpers.AbstractUnmarshallerImpl.getXMLReader(AbstractUnmarshallerImpl.java:106)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:235)
	at javax.xml.bind.JAXB.unmarshal(JAXB.java:223)
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:345)
	at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getCookieConfiguration(RestCrowdClient.java:1017)
	at com.atlassian.crowd.integration.http.CrowdHttpAuthenticatorImpl.getCookieTokenKey(CrowdHttpAuthenticatorImpl.java:214)
	at com.atlassian.crowd.integration.http.CrowdHttpAuthenticatorImpl.getUser(CrowdHttpAuthenticatorImpl.java:48)
	at com.atlassian.crowd.integration.http.CacheAwareCrowdHttpAuthenticator.getUser(CacheAwareCrowdHttpAuthenticator.java:28)
	at com.atlassian.crowd.integration.seraph.v25.CrowdAuthenticator.getUser(CrowdAuthenticator.java:364)
	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:132)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
...
	at java.lang.Thread.run(Thread.java:724)

      Note on fix

      Problem contains 3 parts:

      • Checking cookie.tokenkey value in crowd.properties file and triggering a remote call to the Crowd server when the value is not present
      • creating JAXBContext classes - this is fixed
      • in a newly created Unmarshaller creating new SAXParser (not fixed, creates most of contention)

      Workaround

      Set the cookie.tokenkey to the value configured in the Crowd server. This is detailed in The crowd.properties file. Specifically:

      1. Check the crowd.properties file in Crowd to see if the cookie.tokenkey has been specified. If it has not then it will be the default value of crowd.token_key.
      2. Change the crowd.properties file for every application that is using Crowd to have the value set to either the value from step 1 or the default of crowd.token_key.
        The location of crowd.properties file is different for each application. See each link for more information:

      An example of this configuration:

      application.name                        theplanetarium
application.password                    sauronlosthiscontact
application.login.url                   https://jira.atlassian.com/

crowd.server.url                        http://crowd.atlassian.com:8095/crowd/services/
crowd.base.url                          http://crowd.atlassian.com:8095/crowd/

session.isauthenticated                 session.isauthenticated
session.tokenkey                        session.tokenkey
session.validationinterval              2
session.lastvalidation                  session.lastvalidation

cookie.tokenkey				crowd.token_key

        1. All the Locks.png
          All the Locks.png
          762 kB

            [CWD-3983] XML deserialisation happens in the Crowd client in a non-optimal way, resulting in WebappClassLoader lock congestion

            Pawel Bugalski (Inactive) added a comment -

            dberrueta I was referring to this line:

            return JAXB.unmarshal(method.getResponseBodyAsStream(), returnType);

            in RestExecutor

            which is more or less equivalent to

            JAXBContext.newInstance(returnType).createUnmarshaller().unmarshal(method.getResponseBodyAsStream(), returnType);

            So new JAXBContext and new Unmarshaller are created every time andReceive is called.

            As described here https://jaxb.java.net/guide/Performance_and_thread_safety.html caching JAXBContext and Unmarshaller is a good idea.

            I don't have a good way to reproduce problems caused by this, but you can easily see those problems analyzing
            thread dump jstack_hlmq_27326_20140224_102312_3.txt from https://support.atlassian.com/browse/JSP-183736
            There is a high congestion on one of locks:
            <0x00000005453089d8> (a org.apache.catalina.loader.WebappClassLoader): 0 Thread(s) sleeping, 44 Thread(s) waiting, 1 Thread(s) locking

            This is caused mainly by two costly operations that acquire this lock:

            • in a newly created Unmarshaller new SAXParser is created (it is than cached inside this Unmarshaller instance)
            • while creating JAXBContext classes are accessed by reflection

            Pawel Bugalski (Inactive) added a comment - dberrueta I was referring to this line: return JAXB.unmarshal(method.getResponseBodyAsStream(), returnType); in RestExecutor which is more or less equivalent to JAXBContext.newInstance(returnType).createUnmarshaller().unmarshal(method.getResponseBodyAsStream(), returnType); So new JAXBContext and new Unmarshaller are created every time andReceive is called. As described here https://jaxb.java.net/guide/Performance_and_thread_safety.html caching JAXBContext and Unmarshaller is a good idea. I don't have a good way to reproduce problems caused by this, but you can easily see those problems analyzing thread dump jstack_hlmq_27326_20140224_102312_3.txt from https://support.atlassian.com/browse/JSP-183736 There is a high congestion on one of locks: <0x00000005453089d8> (a org.apache.catalina.loader.WebappClassLoader): 0 Thread(s) sleeping, 44 Thread(s) waiting, 1 Thread(s) locking This is caused mainly by two costly operations that acquire this lock: in a newly created Unmarshaller new SAXParser is created (it is than cached inside this Unmarshaller instance) while creating JAXBContext classes are accessed by reflection

              dberrueta Diego Berrueta
              dcurrie@atlassian.com Dave C
              Affected customers:
              0 This affects my team
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: