Uploaded image for project: 'FishEye'
  1. FishEye
  2. FE-2966

Deadlock in Jetty connection timeout

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Medium
    • 2.5.3
    • 2.5.1
    • None
    • None

    Description

      Thread dumps report:

      Found one Java-level deadlock:
      =============================
      "btpool0-3136":
      waiting to lock monitor 0x00002aac013418f0 (object 0x00002aab3ca97360, a org.mortbay.io.nio.SelectorManager$SelectSet),
      which is held by "btpool0-1 - Acceptor0 ProxyPassingSelectChannelConnector@0.0.0.0:8100"
      "btpool0-1 - Acceptor0 ProxyPassingSelectChannelConnector@0.0.0.0:8100":
      waiting to lock monitor 0x00002aac006aa078 (object 0x00002aab1697b6e0, a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint),
      which is held by "btpool0-3136"

      Java stack information for the threads listed above:
      ===================================================
      "btpool0-3136":
      at org.mortbay.io.nio.SelectorManager$SelectSet.scheduleIdle(SelectorManager.java:782)

      waiting to lock <0x00002aab3ca97360> (a org.mortbay.io.nio.SelectorManager$SelectSet)
      at org.mortbay.io.nio.SelectChannelEndPoint.scheduleIdle(SelectChannelEndPoint.java:159)
      at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:293)
      locked <0x00002aab1697b6e0> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
      at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:544)
      at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571)
      at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1010)
      at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:650)
      at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
      at org.mortbay.util.IO.copy(IO.java:196)
      at org.mortbay.resource.Resource.writeTo(Resource.java:562)
      at org.mortbay.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:755)
      at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:451)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1217)
      at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
      at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25)
      at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:83)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.cenqua.crucible.filters.CrucibleFilter.doFilter(CrucibleFilter.java:141)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.cenqua.fisheye.web.filters.TotalityFilter.doFilter(TotalityFilter.java:188)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:98)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
      at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69)
      at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.cenqua.fisheye.web.filters.CustomIncludableGzipFilter.doFilter(CustomIncludableGzipFilter.java:19)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.cenqua.fisheye.web.filters.ProductInfoFilter.doFilter(ProductInfoFilter.java:36)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.cenqua.fisheye.web.filters.UpfrontFilter.doFilter(UpfrontFilter.java:52)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at com.atlassian.fecru.profiling.ProfilingServletFilter.doFilter(ProfilingServletFilter.java:73)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
      at org.mortbay.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1209)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
      at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      at org.mortbay.jetty.Server.handle(Server.java:326)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
      "btpool0-1 - Acceptor0 ProxyPassingSelectChannelConnector@0.0.0.0:8100":
      at org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:320)
      waiting to lock <0x00002aab1697b6e0> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
      at org.mortbay.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:456)
      at org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint.close(SelectChannelConnector.java:362)
      at org.mortbay.io.nio.SelectChannelEndPoint.idleExpired(SelectChannelEndPoint.java:174)
      at org.mortbay.io.nio.SelectChannelEndPoint$IdleTask.expire(SelectChannelEndPoint.java:489)
      at org.mortbay.thread.Timeout.tick(Timeout.java:137)
      locked <0x00002aab3ca97360> (a org.mortbay.io.nio.SelectorManager$SelectSet)
      at org.mortbay.thread.Timeout.tick(Timeout.java:153)
      at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:753)
      at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
      at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
      at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
      at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
      Found 1 deadlock.

      This appears to be Jetty bug:
      http://jira.codehaus.org/browse/JETTY-1264
      fixed in 6.1.26
      Latest Fisheye version 2.5.1 is running on 6.1.24

      We should upgrade Jetty to help prevent this issue.

      Attachments

        Activity

          People

            jxie Chii
            ganand Gurleen Anand [Atlassian]
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 5h
                5h
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h Time Not Required
                3h