Details
-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
6.4.11
-
6.04
-
10
-
Severity 3 - Minor
-
1
-
Description
Summary
Index corrupted when administrator trigger a shutdown on a node. The index continue to be broken after the restart and JIRA did not automatically recover the index, causing continuous error of FileNotFoundException spamming the log.
Environment
JIRA Data Center 6.4.11
Steps to Reproduce
Restarting the node gracefully (./stop-jira.sh or ./shutdown.sh) while JIRA is performing index update for issue update/change.
Expected Results
No error thrown and index should work after restart.
Actual Results
Index got corrupted before JIRA complete shutting down and unable to recover itself after the restart.
The below exception is thrown in the atlassian-jira.log file:
Caused by: java.io.FileNotFoundException: /usr/local/test/user/atlassian/jira/data/caches/indexes/issues/_skx8_l.del (No such file or directory)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:214)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)
at org.apache.lucene.index.SegmentReader.loadDeletedDocs(SegmentReader.java:158)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677)
at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410)
at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:143)
This means that index entries were supposed to be deleted from an index file, but it doesn't exist. The stack trace is very similar to the one in https://jira.atlassian.com/browse/JRA-41409, but this one died 4 lines later in SegmentReader.get.
Notes
Right after shutdown is triggered shown in the log, the node will hit into InterruptedException error
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at com.atlassian.jira.index.QueueingIndex.perform(QueueingIndex.java:74)
... 34 more
This means that a request to index an issue was sent, but it was rejected, due to shutdown. This stack trace does not indicate index corruption, but increase its possibility.
Workaround
After the node restart, trigger Index Copy manually to recover a working index snapshot from another node.
Attachments
Issue Links
- is cloned from
-
-
- Closed
-
- is related to
-
-
- Closed
-
- mentioned in
-
Page Loading...