Uploaded image for project: 'Crowd Data Center'
  1. Crowd Data Center
  2. CWD-3983

XML deserialisation happens in the Crowd client in a non-optimal way, resulting in WebappClassLoader lock congestion

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • 2.8
    • None
    • None
    • None

      Problem

      If the cookie.tokenkey is not set in crowd.properties, any Atlassian application integrating with it will perform additional calls to the remote Crowd instance to get it from the cookie configuration. In this scenario it's possible an XML deserialisation will happen in the Crowd client in a non-optimal way, resulting in WebappClassLoader lock congestion. This can cause an instance to become unresponsive ("crash").

      Verification

      Taking thread dumps during a problematic period and analysing them will show the below stack traces:

      "catalina-exec-7615" daemon prio=10 tid=0x00002aaae5ba2000 nid=0xa4c waiting for monitor entry [0x0000000066d72000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java)
      	- waiting to lock <0x00000005453089d8> (a org.apache.catalina.loader.WebappClassLoader)
      	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1556)
      	at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
      	at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
      	at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
      	at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
      	at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
      	at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.<init>(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source)
      	at javax.xml.bind.helpers.AbstractUnmarshallerImpl.getXMLReader(AbstractUnmarshallerImpl.java:106)
      	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:235)
      	at javax.xml.bind.JAXB.unmarshal(JAXB.java:223)
      	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:345)
      	at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getCookieConfiguration(RestCrowdClient.java:1017)
      	at com.atlassian.crowd.integration.http.CrowdHttpAuthenticatorImpl.getCookieTokenKey(CrowdHttpAuthenticatorImpl.java:214)
      	at com.atlassian.crowd.integration.http.CrowdHttpAuthenticatorImpl.getUser(CrowdHttpAuthenticatorImpl.java:48)
      	at com.atlassian.crowd.integration.http.CacheAwareCrowdHttpAuthenticator.getUser(CacheAwareCrowdHttpAuthenticator.java:28)
      	at com.atlassian.crowd.integration.seraph.v25.CrowdAuthenticator.getUser(CrowdAuthenticator.java:364)
      	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:132)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      ...
      	at java.lang.Thread.run(Thread.java:724)
      

      Note on fix

      Problem contains 3 parts:

      • Checking cookie.tokenkey value in crowd.properties file and triggering a remote call to the Crowd server when the value is not present
      • creating JAXBContext classes - this is fixed
      • in a newly created Unmarshaller creating new SAXParser (not fixed, creates most of contention)

      Workaround

      Set the cookie.tokenkey to the value configured in the Crowd server. This is detailed in The crowd.properties file. Specifically:

      1. Check the crowd.properties file in Crowd to see if the cookie.tokenkey has been specified. If it has not then it will be the default value of crowd.token_key.
      2. Change the crowd.properties file for every application that is using Crowd to have the value set to either the value from step 1 or the default of crowd.token_key.
        The location of crowd.properties file is different for each application. See each link for more information:

      An example of this configuration:

      application.name                        theplanetarium
      application.password                    sauronlosthiscontact
      application.login.url                   https://jira.atlassian.com/
      
      crowd.server.url                        http://crowd.atlassian.com:8095/crowd/services/
      crowd.base.url                          http://crowd.atlassian.com:8095/crowd/
      
      session.isauthenticated                 session.isauthenticated
      session.tokenkey                        session.tokenkey
      session.validationinterval              2
      session.lastvalidation                  session.lastvalidation
      
      cookie.tokenkey				crowd.token_key
      

              dberrueta Diego Berrueta
              dcurrie@atlassian.com Dave C
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: