Details
-
Bug
-
Resolution: Fixed
-
Highest
-
6.4.12, 7.1.7, 7.3.0, 7.2.7, 7.4.0, 7.5.0, 7.6.0, 7.7.0
-
6.04
-
27
-
Severity 1 - Critical
-
119
-
Description
Summary
The JIRA Lucene index is based on the data in DB, effectively it's a search index. Any problems with Lucene index will cause problem with search results, JIRA Dashboards, or JIRA Software Dashboards. Lucene Index might get corrupted under high load and ungraceful shutdown or crash of JIRA/JVM.
Expected Results
It recovers from transaction log or nearest checkpoint.
Actual Results
Lucene data is corrupted and JIRA doesn't function properly.
2016-05-17 04:48:49,064 http-bio-8080-exec-16 ERROR admin288x38453x8 1k54nbw 127.0.0.1 /browse/PP [jira.projects.web.BrowseProject] Tab panel 'com.atlassian.jira.jira-projects-plugin:summary-panel' failed to render.
com.atlassian.jira.issue.index.SearchUnavailableException: com.atlassian.jira.util.RuntimeIOException: java.io.FileNotFoundException: /atlassian/application-data/jira/caches/indexes/issues/_nfbu.cfs (No such file or directory)
Notes
Plugins continue to operate After consistencyChecker is Run
- IndexWriters are being shutdown and closed in consistencyChecker.destroy(ServletContextProvider.getServletContext()).
- The problem is that we stop plugins after it. So SD Automation is still running and it creates a new IssueIndexer.
- This also explains, why we see the following in the end of the log file:
12-Jun-2017 15:03:42.169 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [ISSUE-indexQueue:thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) com.atlassian.jira.index.QueueingIndex$Task.index(QueueingIndex.java:142) com.atlassian.jira.index.QueueingIndex$Task.run(QueueingIndex.java:111) java.lang.Thread.run(Thread.java:745)
The second instance of IssueIndexer thread isn't stopped by anybody. So, it
- Is still running in the end of shutdown;
- Doesn't flush the changes to index file.
Force Kill via stop-jira.sh
- By default bin/stop-jira.sh script forces JIRA shutdown after 20 seconds:
exec $PRGDIR/shutdown.sh 20 -force $@
from bin/catalina.sh
if [ $FORCE -eq 1 ]; then .. if [ -f "$CATALINA_PID" ]; then PID=`cat "$CATALINA_PID"` echo "Killing Tomcat with the PID: $PID" kill -9 $PID
- JIRA uses Lucene 3.3.0.
- Due to inefficiency in JIRA 6 plugin system, shutting down JIRA instance could take up to 10 minutes.
- Segments File description http://lucene.apache.org/core/3_3_0/fileformats.html#Segments%20File
- Possibly caused by https://issues.apache.org/jira/browse/LUCENE-3418
Workaround
- Enable Index Recovery, please see Backing up and recovering your index
- Fix the corrupted indexes manually, see KB article: Lucene index corruption
- For JIRA 7.2.10, or later (on the 7.2 line, see
JRASERVER-65655) the corrupted indexes are fixed automatically. There will be some data loss in Lucene index after recovery. See KB Lucene index corruption for details.
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-
- is related to
-
JRASERVER-65655 Fix corrupted index during startup
- Closed
- relates to
-
-
- Closed
-
-
-
- Gathering Impact
-
-
RUM-74 Loading...
-
- is blocked by
-
- links to
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-