Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-70651

Invalid rows in nodeassocation will cause a cluster-wide outage

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • High
    • 9.11.1
    • 6.4.14, 7.2.15, 7.6.16, 7.13.12, 8.5.3, 8.13.7, 8.20.16, 9.4.6, 9.11.0
    • Project - Actions

    Description

      Issue Summary

      All nodes in a Jira Data Center cluster will become inaccessible if there are rows in nodeassocation that refer to a project ID that does not exist in the project table ( aka project schemes for a project that does not exist ).

      Steps to Reproduce

      1. Not reliably reproducible, we only have reports of this occurring after the fact.
      2. However, you can create a row manually in nodeassociation using a project id value that doesn't exist in the project table to trigger this behavior

      Expected Results

      If this occurs, for whatever reason, Jira at least remains accessible

      Actual Results

      Jira becomes inaccessible, throwing 500 errors in the user interface until the rows are removed and every node is restarted

      The below exception is thrown in the atlassian-jira.log file:

      2020-02-03 17:51:15,860 http-nio-8080-exec-9 url:/jira/ url:/jira/secure/MyJiraHome.jspa url:/jira/secure/
      Dashboard.jspa ERROR      [o.a.c.c.C.[.[.[/jira].[jsp.default_jsp]] Servlet.service() for servlet [jsp.default_jsp] in context with path [/jira] threw exception
      java.lang.RuntimeException: javax.servlet.ServletException: java.lang.RuntimeException: javax.servlet.ServletException: java.lang.RuntimeException: com.atlassian.cache.CacheException: java.lang.NullPointerException
      	at com.atlassian.web.servlet.plugin.DynamicAuthorizationServletForwarder.forward(DynamicAuthorizationServletForwarder.java:55)
      	at com.atlassian.web.servlet.plugin.SanitizingServletForwarder.forward(SanitizingServletForwarder.java:32)
      	at com.atlassian.web.servlet.plugin.RememberingServletForwarder.forward(RememberingServletForwarder.java:51)
      	at com.atlassian.web.servlet.plugin.ResolvingServletForwarder.forward(ResolvingServletForwarder.java:36)
      	at jsp.default_jsp._jspService(default_jsp.java:68)
      	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
      	... 57 filtered
      ...
          Caused by: com.atlassian.cache.CacheException: java.lang.NullPointerException
          	at com.atlassian.cache.ehcache.DelegatingCachedReference.get(DelegatingCachedReference.java:77) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.jira.cluster.cache.ehcache.BlockingParallelCacheReplicator.runDeferred(BlockingParallelCacheReplicator.java:172) [classes/:?]
          	at com.atlassian.jira.cache.DeferredReplicationCachedReference.get(DeferredReplicationCachedReference.java:28) [classes/:?]
          	at com.atlassian.jira.project.CachingProjectManager.getProjects(CachingProjectManager.java:108) [classes/:?]
          	at com.atlassian.jira.project.CachingProjectManager.getProjectObjects(CachingProjectManager.java:128) [classes/:?]
          	at com.atlassian.jira.security.DefaultPermissionManager.hasProjects(DefaultPermissionManager.java:254) [classes/:?]
          	at com.atlassian.jira.security.ApplicationRequiredPermissionManager.lambda$hasProjects$11(ApplicationRequiredPermissionManager.java:138) [classes/:?]
          	at com.atlassian.jira.security.ApplicationRequiredPermissionManager.checkUserHasApplicationOrFalse(ApplicationRequiredPermissionManager.java:187) [classes/:?]
          	at com.atlassian.jira.security.ApplicationRequiredPermissionManager.hasProjects(ApplicationRequiredPermissionManager.java:138) [classes/:?]
          	at com.atlassian.jira.security.PublicAccessPermissionManager.lambda$hasProjects$11(PublicAccessPermissionManager.java:142) [classes/:?]
          	at com.atlassian.jira.security.PublicAccessPermissionManager.checkPublicAccessEnabledOrDelegate(PublicAccessPermissionManager.java:191) [classes/:?]
          	at com.atlassian.jira.security.PublicAccessPermissionManager.hasProjects(PublicAccessPermissionManager.java:142) [classes/:?]
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [?:1.8.0_191]
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [?:1.8.0_191]
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_191]
          	at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_191]
          	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38) [classes/:?]
          	at com.sun.proxy.$Proxy15.hasProjects(Unknown Source) [?:?]
          	at com.atlassian.jira.dashboard.permission.JiraGadgetPermissionManager.hasProjectsPermission(JiraGadgetPermissionManager.java:167) [classes/:?]
          	... 615 more
          Caused by: java.lang.NullPointerException
          	at java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:469) [?:1.8.0_191]
          	at java.util.TimSort.binarySort(TimSort.java:296) [?:1.8.0_191]
          	at java.util.TimSort.sort(TimSort.java:239) [?:1.8.0_191]
          	at java.util.Arrays.sort(Arrays.java:1512) [?:1.8.0_191]
          	at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:348) [?:1.8.0_191]
          	at java.util.stream.Sink$ChainedReference.end(Sink.java:258) [?:1.8.0_191]
          	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) [?:1.8.0_191]
          	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) [?:1.8.0_191]
          	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) [?:1.8.0_191]
          	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) [?:1.8.0_191]
          	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:1.8.0_191]
          	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) [?:1.8.0_191]
          	at com.atlassian.jira.project.ProjectCache.refreshCategoryProjectMappings(ProjectCache.java:208) [classes/:?]
          	at com.atlassian.jira.project.ProjectCache.init(ProjectCache.java:94) [classes/:?]
          	at com.atlassian.jira.project.ProjectCache.<init>(ProjectCache.java:81) [classes/:?]
          	at com.atlassian.jira.project.CachingProjectManager.lambda$new$0(CachingProjectManager.java:40) [classes/:?]
          	at com.atlassian.cache.ehcache.EhCacheManager$SupplierAdapter.load(EhCacheManager.java:260) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.cache.ehcache.EhCacheManager$SupplierAdapter.load(EhCacheManager.java:247) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.cache.ehcache.wrapper.ValueProcessorAtlassianCacheLoaderDecorator.load(ValueProcessorAtlassianCacheLoaderDecorator.java:26) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.cache.ehcache.LoadingCache.getFromLoader(LoadingCache.java:133) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.cache.ehcache.SynchronizedLoadingCacheDecorator.synchronizedLoad(SynchronizedLoadingCacheDecorator.java:29) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.cache.ehcache.LoadingCache.loadValueAndReleaseLock(LoadingCache.java:101) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.cache.ehcache.LoadingCache.get(LoadingCache.java:80) [atlassian-cache-ehcache-3.2.0.jar:?]
          	at com.atlassian.cache.ehcache.DelegatingCachedReference.get(DelegatingCachedReference.java:73) [atlassian-cache-ehcache-3.2.0.jar:?]
          	... 633 more
          , referer=Unknown, servletErrorMessage=}
      

      Workaround

      Remove the invalid rows as described in 500 CacheException error thrown during upgrade/restore of Jira server, shut down all nodes, and start them up again as database values may remain in Jira's cache

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jcurry Jeff Curry
              Votes:
              32 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: