Details
-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
7.2.13
-
7.02
-
9
-
Severity 1 - Critical
-
3
-
Description
Problem Statement
When an app-linked resource is not responsive in a way that holds open a HTTP socket, the calling functionality hangs indefinitely.
We have observed this at least, with FishEye / Crucible integration. Based on the stack trace, we see no reason to believe it could not happen for other integrations also
Expected Behavior
Jira is resilient to failures in external applications connected to it. If the HTTP call does not complete within in a reasonable time frame, the request is terminated and the process is allowed to continue.
Actual Behavior
Performance issues in Development Status providers - Fisheye/Crucible, Bitbucket... - may lead to an outage as multiple threads request information from these sources at the same time - when loading issues, for example - and we never time out the requests.
Example thread - loading issue dev panel
"ajp-nio-11016-exec-454" #172419 daemon prio=5 os_prio=0 tid=0x00007f7800166800 nid=0x10c3 runnable [0x00007f75db5e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) ... at com.atlassian.jirafisheyeplugin.rest.FishEyeRestApiManagerImpl.callFisheye(FishEyeRestApiManagerImpl.java:178) at com.atlassian.jirafisheyeplugin.rest.FishEyeRestApiManagerImpl.callFisheye(FishEyeRestApiManagerImpl.java:127) at com.atlassian.jirafisheyeplugin.domain.fisheye.FishEyeInstanceImpl.isCrossRepoCapable(FishEyeInstanceImpl.java:104) at com.atlassian.jirafisheyeplugin.domain.fisheye.ChangeSetManagerImpl.searchForChangeSets(ChangeSetManagerImpl.java:183) at com.atlassian.jirafisheyeplugin.domain.fisheye.ChangeSetManagerImpl.getChangesetsForIssue(ChangeSetManagerImpl.java:105) at com.atlassian.jirafisheyeplugin.domain.fisheye.ChangeSetManagerImpl.getChangesetsForIssue(ChangeSetManagerImpl.java:90) at com.atlassian.jirafisheyeplugin.web.issuetabpanel.fisheye.FishEyeIssueTabPanel.createActions(FishEyeIssueTabPanel.java:108) at com.atlassian.jirafisheyeplugin.web.issuetabpanel.AbstractFishEyeCrucibleIssueTabPanel.getActions(AbstractFishEyeCrucibleIssueTabPanel.java:107) at com.atlassian.jira.plugin.issuetabpanel.IssueTabPanelInvokerImpl$2.call(IssueTabPanelInvokerImpl.java:77) at com.atlassian.jira.plugin.issuetabpanel.IssueTabPanelInvokerImpl$2.call(IssueTabPanelInvokerImpl.java:71) at com.atlassian.ozymandias.SafePluginPointAccess.call(SafePluginPointAccess.java:187) at com.atlassian.jira.plugin.issuetabpanel.IssueTabPanelInvokerImpl.invokeGetActions(IssueTabPanelInvokerImpl.java:71) at com.atlassian.jira.issue.tabpanels.AllTabPanel.getActions(AllTabPanel.java:34) at com.atlassian.jira.plugin.issuetabpanel.IssueTabPanel3Adaptor.getActions(IssueTabPanel3Adaptor.java:37) at com.atlassian.jira.plugin.issuetabpanel.IssueTabPanelInvokerImpl$2.call(IssueTabPanelInvokerImpl.java:77) at com.atlassian.jira.plugin.issuetabpanel.IssueTabPanelInvokerImpl$2.call(IssueTabPanelInvokerImpl.java:71) at com.atlassian.ozymandias.SafePluginPointAccess.call(SafePluginPointAccess.java:187) at com.atlassian.jira.plugin.issuetabpanel.IssueTabPanelInvokerImpl.invokeGetActions(IssueTabPanelInvokerImpl.java:71) ... at com.atlassian.jira.plugin.viewissue.ActivityBlockViewIssueContextProvider.getContextMap(ActivityBlockViewIssueContextProvider.java:102) at com.atlassian.jira.plugin.webfragment.CacheableContextProviderDecorator.initContextMap(CacheableContextProviderDecorator.java:63) at com.atlassian.jira.plugin.webfragment.CacheableContextProviderDecorator.getContextMap(CacheableContextProviderDecorator.java:41) at com.atlassian.jira.plugin.webfragment.contextproviders.MultiContextProvider.getContextMap(MultiContextProvider.java:86) at com.atlassian.jira.plugin.bigpipe.BigPipeWebPanelModuleDescriptor$BigPipeDelegatingWebPanel.getHtml(BigPipeWebPanelModuleDescriptor.java:251) at com.atlassian.jira.web.component.ModuleWebComponentImpl.renderModule(ModuleWebComponentImpl.java:81) at com.atlassian.jira.web.component.ModuleWebComponentImpl.access$000(ModuleWebComponentImpl.java:30) ... at com.atlassian.jira.issue.util.IssueWebPanelRenderUtil.renderHeadlessPanel(IssueWebPanelRenderUtil.java:90) at com.atlassian.jira.components.issueviewer.viewissue.webpanel.WebPanelBeanBuilder.buildWithHtml(WebPanelBeanBuilder.java:51) at com.atlassian.jira.components.issueviewer.viewissue.webpanel.WebPanelMapperUtilImpl.mapAndRenderPanels(WebPanelMapperUtilImpl.java:78) at com.atlassian.jira.components.issueviewer.viewissue.webpanel.WebPanelMapperUtilImpl.create(WebPanelMapperUtilImpl.java:64) at com.atlassian.jira.components.issueviewer.viewissue.webpanel.WebPanelMapperUtilImpl.create(WebPanelMapperUtilImpl.java:52) ... at com.atlassian.jira.projects.issuenavigator.DefaultIssueRenderService.render(DefaultIssueRenderService.java:97) at com.atlassian.jira.projects.issuenavigator.webaction.ProjectIssueNavigatorAction.doIssueViewWithSidebar(ProjectIssueNavigatorAction.java:191) ...
Example thread - blocking startup
"JIRA-Bootstrap" #16 daemon prio=5 os_prio=0 tid=0x00007f4f1cb64800 nid=0x7f41 runnable [0x00007f4f5c8d1000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) ... at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1300) ... at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:101) ... at com.atlassian.jirafisheyeplugin.rest.FishEyeRestApiManagerImpl.callFisheye(FishEyeRestApiManagerImpl.java:178) at com.atlassian.jirafisheyeplugin.rest.FishEyeRestApiManagerImpl.callFisheye(FishEyeRestApiManagerImpl.java:127) at com.atlassian.jirafisheyeplugin.domain.fisheye.FishEyeManagerImpl.getProjectsImpl(FishEyeManagerImpl.java:402) at com.atlassian.jirafisheyeplugin.domain.fisheye.FishEyeManagerImpl.refreshCrucibleProjects(FishEyeManagerImpl.java:215) at com.atlassian.jirafisheyeplugin.domain.fisheye.FishEyeManagerImpl.refresh(FishEyeManagerImpl.java:269) at com.atlassian.jirafisheyeplugin.config.RefreshManagerImpl.refreshRefreshables(RefreshManagerImpl.java:161) at com.atlassian.jirafisheyeplugin.config.RefreshManagerImpl.refreshAll(RefreshManagerImpl.java:79) - locked <0x00000003d4b8c3a8> (a com.atlassian.jirafisheyeplugin.config.RefreshManagerImpl) at com.atlassian.jirafisheyeplugin.config.RefreshManagerImpl.onStart(RefreshManagerImpl.java:93) ... at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:100)
Steps to Reproduce
TBD
Workaround
Workaround 1
- Set the DNS host file for fisheye server :
127.0.0.1 fisheye.server.com
- If fisheye server is behind the proxy, set timeout at proxy level (or block request)
One FishEye / Cricible is back online, you can remove the host file entry and restart the Jira service.
Workaround 2
- Disable the jirafisheyeplugin plugin via database query. This requires application restart.
insert into pluginstate values('com.atlassian.jirafisheyeplugin','false');
- Make HTTP call fail fast, a couple of possible options:
Note: The fisheye plugin will get automatically enabled again after startup completes. So, if the node reboots, the SQL will need to be run again followed by a restart. This may mean Workaround 1 is the preferred option
2022-08-15 02:30:38,226+0200 Caesium-1-4 INFO anonymous [c.a.j.p.software.monitor.ApplicationPluginsEnablerJobRunner] Attempting to enable the following application plugins: [com.atlassian.jirafisheyeplugin]