Summary

Method com.atlassian.jira.index.summary.DefaultIndexSummarizer#getCountOfIssuesInIndex forces issueSearcher to be closed each execution. If IndexSearcher.close() is called it might result in closing the underneath reader and causing JVM crash with SIGSEGV under heavy load.

Expected Results

Method getCountOfIssuesInIndex() will not close issueSearcher

Actual Results

Method getCountOfIssuesInIndex() closes issueSearcher
You may see the following exception:

com.atlassian.jira.index.DelayCloseable$AlreadyClosedException: java.io.IOException
        at com.atlassian.jira.index.DelayCloseable$Helper.close(DelayCloseable.java:77)
        at com.atlassian.jira.index.DelayCloseSearcher.close(DelayCloseSearcher.java:57)
        at com.atlassian.jira.issue.index.SearcherCache.closeSearcher(SearcherCache.java:79)
        at com.atlassian.jira.issue.index.SearcherCache.closeSearchers(SearcherCache.java:60)
        at com.atlassian.jira.issue.index.DefaultIndexManager.flushThreadLocalSearchers(DefaultIndexManager.java:1020)
        at com.atlassian.jira.util.searchers.ThreadLocalSearcherCache.resetSearchers(ThreadLocalSearcherCache.java:31)
        ... 20 filtered
        at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
        ... 26 filtered
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException
        at com.atlassian.jira.index.DelayCloseable$AlreadyClosedException.<init>(DelayCloseable.java:37)
        ... 57 more

Notes

• Under heavy load on Jira and execution of getCountOfIssuesInIndex() may close index that is still being read by another thread and as a result cause  SIGSEGV like this:

  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGSEGV (0xb) at pc=0x00007f66338faaea, pid=66449, tid=0x00007f4dd99f1700
  #
  # JRE version: Java(TM) SE Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
  # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 )
  # Problematic frame:
  # J 8554 C2 org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte()B (81 bytes) @ 0x00007f66338faaea [0x00007f66338faaa0+0x4a]
  

  • More information why this is happening can be found here https://issues.apache.org/jira/browse/LUCENE-6943?focusedCommentId=15070960&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15070960 
• Another example of SIGSEGV:

  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGSEGV (0xb) at pc=0x00007fb0f16fa390, pid=63009, tid=0x00007f9fb94d2700
  #
  # JRE version: Java(TM) SE Runtime Environment (8.0_152-b16) (build 1.8.0_152-b16)
  # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode linux-amd64 )
  # Problematic frame:
  # v  ~StubRoutines::jbyte_disjoint_arraycopy
  
  Stack: [0x00007fc9d9dc6000,0x00007fc9d9ec7000],  sp=0x00007fc9d9ebfff0,  free space=999k
  Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
  v  ~StubRoutines::jbyte_disjoint_arraycopy
  J 109528 C2 org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal([BII)V (55 bytes) @ 0x00007fdc036f1882 [0x00007fdc036f14a0+0x3e2]
  J 23635 C2 org.apache.lucene.store.BufferedIndexInput.readByte()B (32 bytes) @ 0x00007fdbfb5b24b4 [0x00007fdbfb5b2400+0xb4]
  J 31429 C2 org.apache.lucene.store.DataInput.readVInt()I (132 bytes) @ 0x00007fdbff486510 [0x00007fdbff4864e0+0x30]
  

  • example at Oracle tracker: https://bugs.openjdk.java.net/browse/JDK-8179671

Workaround

None