all JIRA threads is blocked waiting for cache getProjectContextIdsForQuery

Summary

JIRA is inaccessible and you get This site can’t be reached error. Almost all incoming http threads except one are blocked waiting for cache getProjectContextIdsForQuery to be populated.

Steps to Reproduce

1. Restart JIRA
2. Initiate large number of call loading data from getProjectContextIdsForQuery cache, eg: URL: /secure/BrowseProject.jspa
3. Check that JIRA is not accessible after some time.

Expected Results

JIRA is working fine

Actual Results

• Almost all threads are blocked waiting for cache getProjectContextIdsForQuery cache, example:

  http-bio-80-exec-88" #306 daemon prio=5 os_prio=0 tid=0x0000000032d35000 nid=0x1cb4 waiting on condition [0x000000025e419000]
     java.lang.Thread.State: WAITING (parking)
  	at sun.misc.Unsafe.park(Native Method)
  	- parking to wait for  <0x00000007b7c020f0> (a com.google.common.util.concurrent.AbstractFuture$Sync)
  ...
  	at com.atlassian.cache.memory.DelegatingCache$DelegatingLoadingCache.get(DelegatingCache.java:324)
  	at com.atlassian.cache.compat.delegate.DelegatingCache.get(DelegatingCache.java:37)
  	at com.atlassian.greenhopper.service.query.QueryToProjectMapper.getProjectContextIdsForQuery(QueryToProjectMapper.java:78)
  	at com.atlassian.greenhopper.service.query.QueryServiceImpl.getProjectsForRapidView(QueryServiceImpl.java:116)
  	at com.atlassian.greenhopper.service.query.QueryServiceImpl.getExplicitProjectsForRapidViewFilterQuery(QueryServiceImpl.java:103)
  	at com.atlassian.greenhopper.service.rapid.ProjectRapidViewServiceImpl.findProjectsByRapidView(ProjectRapidViewServiceImpl.java:57)
  	at com.atlassian.greenhopper.service.rapid.ProjectRapidViewServiceImpl.findRapidViewsByProject(ProjectRapidViewServiceImpl.java:40)
  	at com.atlassian.greenhopper.web.sidebar.ProjectBoardsHelper.getProjectBoards(ProjectBoardsHelper.java:53)
  	at com.atlassian.greenhopper.web.sidebar.BoardScopeFilterContextProviderImpl.getContext(BoardScopeFilterContextProviderImpl.java:37)
  	at com.atlassian.greenhopper.web.sidebar.AgileBoardScopeFilterContextProvider.getContext(AgileBoardScopeFilterContextProvider.java:27)
  	at com.atlassian.jira.projects.context.ProjectContextPopulator.populateWithProject(ProjectContextPopulator.java:77)
  	at com.atlassian.jira.projects.sidebar.content.ProjectContentRendererImpl.render(ProjectContentRendererImpl.java:35)
  	at com.atlassian.jira.projects.sidebar.ProjectSidebarRendererImpl.render(ProjectSidebarRendererImpl.java:78)
  	at com.atlassian.greenhopper.sidebar.ProjectSidebarRendererAdapterImpl.render(ProjectSidebarRendererAdapterImpl.java:21)
  	at com.atlassian.greenhopper.sidebar.ProjectSidebarRenderer.render(ProjectSidebarRenderer.java:45)
  	at com.atlassian.greenhopper.web.sidebar.BoardSidebarRenderer.getProjectSidebarHtml(BoardSidebarRenderer.java:157)
  	at com.atlassian.greenhopper.web.sidebar.BoardSidebarRenderer.getSidebarHtml(BoardSidebarRenderer.java:88)
  	at com.atlassian.greenhopper.web.rapid.RapidBoardAction.getSidebarHtml(RapidBoardAction.java:307)
  ...
  

• 1 thread is populating the cache getProjectContextIdsForQuery and same thread meanwhile busy doing other slow task, one of possible examples:

  "http-bio-80-exec-121" #345 daemon prio=5 os_prio=0 tid=0x00000000342d3800 nid=0x1130 runnable [0x0000000262a98000]
     java.lang.Thread.State: RUNNABLE
  	at java.net.SocketInputStream.socketRead0(Native Method)
  ...
          com.atlassian.greenhopper.service.query.FieldPossibleValuesAdapterImpl.possibleValuesForIssueInput(FieldPossibleValuesAdapterImpl.java:21)
  	at com.atlassian.greenhopper.service.query.FieldPossibleValues.getProjectIdsForQuery(FieldPossibleValues.java:38)
  ...
  	at com.atlassian.cache.compat.delegate.DelegatingCache.get(DelegatingCache.java:37)
  	at com.atlassian.greenhopper.service.query.QueryToProjectMapper.getProjectContextIdsForQuery(QueryToProjectMapper.java:78)
  	at com.atlassian.greenhopper.service.query.QueryServiceImpl.getProjectsForRapidView(QueryServiceImpl.java:116)
  	at com.atlassian.greenhopper.service.query.QueryServiceImpl.getExplicitProjectsForRapidViewFilterQuery(QueryServiceImpl.java:103)
  	at com.atlassian.greenhopper.service.rapid.ProjectRapidViewServiceImpl.findProjectsByRapidView(ProjectRapidViewServiceImpl.java:57)
  	at com.atlassian.greenhopper.service.rapid.ProjectRapidViewServiceImpl.findRapidViewsByProject(ProjectRapidViewServiceImpl.java:40)
  	at com.atlassian.greenhopper.web.sidebar.ProjectBoardsHelper.getProjectBoards(ProjectBoardsHelper.java:53)
  	at com.atlassian.greenhopper.web.sidebar.BoardScopeFilterContextProviderImpl.getContext(BoardScopeFilterContextProviderImpl.java:37)
  	at com.atlassian.greenhopper.web.sidebar.AgileBoardScopeFilterContextProvider.getContext(AgileBoardScopeFilterContextProvider.java:27)
  	at com.atlassian.jira.projects.context.ProjectContextPopulator.populateWithProject(ProjectContextPopulator.java:77)
  ...
  

Known slow tasks causing slow getProjectContextIdsForQuery population

• loading data for getVersionsByName cache from DB which is slow for large number of version:

  "http-bio-80-exec-121" #345 daemon prio=5 os_prio=0 tid=0x00000000342d3800 nid=0x1130 runnable [0x0000000262a98000]
     java.lang.Thread.State: RUNNABLE
  	at java.net.SocketInputStream.socketRead0(Native Method)
  ...
  	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
  ...
  	at org.ofbiz.core.entity.GenericDelegator.findAll(GenericDelegator.java:802)
  	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findAll(DefaultOfBizDelegator.java:361)
  ...
  	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
  ...
  	at com.atlassian.jira.project.version.CachingVersionStore.getVersionsByName(CachingVersionStore.java:148)
  	at com.atlassian.jira.project.version.DefaultVersionManager.getVersionsByName(DefaultVersionManager.java:577)
  	at com.atlassian.jira.jql.resolver.VersionResolver.getIdsFromName(VersionResolver.java:30)
  ...
          com.atlassian.greenhopper.service.query.FieldPossibleValuesAdapterImpl.possibleValuesForIssueInput(FieldPossibleValuesAdapterImpl.java:21)
  	at com.atlassian.greenhopper.service.query.FieldPossibleValues.getProjectIdsForQuery(FieldPossibleValues.java:38)
  ...
  	at com.atlassian.cache.compat.delegate.DelegatingCache.get(DelegatingCache.java:37)
  	at com.atlassian.greenhopper.service.query.QueryToProjectMapper.getProjectContextIdsForQuery(QueryToProjectMapper.java:78)
  	at com.atlassian.greenhopper.service.query.QueryServiceImpl.getProjectsForRapidView(QueryServiceImpl.java:116)
  	at com.atlassian.greenhopper.service.query.QueryServiceImpl.getExplicitProjectsForRapidViewFilterQuery(QueryServiceImpl.java:103)
  	at com.atlassian.greenhopper.service.rapid.ProjectRapidViewServiceImpl.findProjectsByRapidView(ProjectRapidViewServiceImpl.java:57)
  	at com.atlassian.greenhopper.service.rapid.ProjectRapidViewServiceImpl.findRapidViewsByProject(ProjectRapidViewServiceImpl.java:40)
  	at com.atlassian.greenhopper.web.sidebar.ProjectBoardsHelper.getProjectBoards(ProjectBoardsHelper.java:53)
  	at com.atlassian.greenhopper.web.sidebar.BoardScopeFilterContextProviderImpl.getContext(BoardScopeFilterContextProviderImpl.java:37)
  	at com.atlassian.greenhopper.web.sidebar.AgileBoardScopeFilterContextProvider.getContext(AgileBoardScopeFilterContextProvider.java:27)
  	at com.atlassian.jira.projects.context.ProjectContextPopulator.populateWithProject(ProjectContextPopulator.java:77)
  ...
  

  See JRA-62666
  This is fixed in recent version of JIRA

Notes

Verification

Taking thread dumps as per Generate a Thread Dump during problematic periods of slowness, and searching for the below stack traces can indicate this is a problem:

grep -c 'http-bio-80-exec' thread_dump1.txt 
150

grep 'http-bio-80-exec' thread_dump1.txt | grep -c 'waiting on condition' 
145

grep -A 20 'http-bio-80-exec' thread_dump1.txt | grep -c 'getProjectContextIdsForQuery' 
145

grep 'http-bio-80-exec' thread_dump1.txt | grep runnable 
"http-bio-80-exec-37" #255 daemon prio=5 os_prio=0 tid=0x000000002b37d000 nid=0x1c48 runnable

Other notes

This actual bug is not caused by large number of version, but happy assumption in cache loader getProjectContextIdsForQuery that it will take feasible amount of time
to resolve queries
Problem with large number of versions is mitigated in JRA-62666 bug and fixed in JIRA 7.1.2 and 6.4.14

Workaround

Applicable for JIRA 6.4.x, From JRA-45178
Use the following steps below.

1. Disable the board-scope-filter module in JIRA Agile Add-on (board-scope-filter module tells the sidebar the list of boards to show on the board dropdown as well as the board that is currently selected.). This will disable com.atlassian.greenhopper.web.sidebar.AgileBoardScopeFilterContextProvider.getContext method.
   1. Navigate through Administration menu > Add-ons > Manage Add-ons > Select 'JIRA Agile'
   2. Expand the JIRA Agile Add-on modules
   3. Find the 'board-scope-filter' module and disable it.
2. Disable Project Navigation (Sidebar project navigation) for all users: Administration Menu > System > Project Navigation