Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-44608

Error while re-indexing the DateOfFirstResponseCFType in the JIRA Charting Plugin

XMLWordPrintable

      NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

      Summary

      • After upgrading from JIRA 5.2.111 to 6.4.8 there are Errors in the logs during the Lock JIRA and rebuild re-index with the JIRA Charting Plugin enabled.
      • There are no visible errors on the JIRA UI.
      • The re-index finishes successfully bu there are errors in the logs
      • Here is the errors from the logs : 
        2015-07-22 10:47:43,844 IssueIndexer:thread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 94% complete. Current index: Issue
2015-07-22 10:47:44,522 IssueIndexer:thread-6 ERROR xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [core.entity.jdbc.SQLProcessor] !!! ABANDONED xxxxxx DETECTED !!!
	This probably means that somebody forgot to close an EntityListIterator.
	Connection: DelegatingConnectionImpl[connectionPoolInfo=ConnectionPoolInfo{maxSize=30, minSize=2, initialSize=null, maxIdle=30, maxWait=60000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery='select 1', minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=null, validationQueryTimeout=null, defaultCatalog=null},sqlConnectionInterceptor=org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor@29d49e9a]
	SQL: select min( act.CREATED) from jiraaction act , jiraissue it where act.issueid = ?  and act.issueid = it.ID and act.actiontype = 'comment'  and it.REPORTER != act.AUTHOR
2015-07-22 10:47:44,524 IssueIndexer:thread-6 ERROR xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor] Unexpected exception from SQL connection interceptor callback: onConnectionReplaced([DelegatingConnectionImpl[connectionPoolInfo=ConnectionPoolInfo{maxSize=30, minSize=2, initialSize=null, maxIdle=30, maxWait=60000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery='select 1', minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=null, validationQueryTimeout=null, defaultCatalog=null},sqlConnectionInterceptor=org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor@29d49e9a], ConnectionPoolStateImpl[timeToBorrowNanos=0,borrowCount=2,connectionPoolInfo=ConnectionPoolInfo{maxSize=30, minSize=2, initialSize=null, maxIdle=30, maxWait=60000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery='select 1', minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=null, validationQueryTimeout=null, defaultCatalog=null}]])
java.lang.IllegalStateException: The OpTimer has been re-used.  end() can only be called once!
	at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:142)
	at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:99)
	at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:117)
	at com.atlassian.jira.ofbiz.InstrumentedSQLInterceptor.onConnectionReplaced(InstrumentedSQLInterceptor.java:39)
	at com.atlassian.jira.ofbiz.ChainedSQLInterceptor.onConnectionReplaced(ChainedSQLInterceptor.java:75)
	at org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor.onConnectionReplaced(SafeDelegatingSqlConnectionInterceptor.java:47)
	at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker$DelegatingConnectionImpl.close(ConnectionTracker.java:99)
	at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedConnection(ConnectionGuard.java:124)
	at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessor(ConnectionGuard.java:112)
	at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessors(ConnectionGuard.java:150)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:390)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:544)
	at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:960)
	at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:883)
	at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:730)
	at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:150)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:901)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:879)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findByAnd(DefaultOfBizDelegator.java:110)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findByAnd(WrappingOfBizDelegator.java:91)
	at com.atlassian.jira.issue.worklog.OfBizWorklogStore.getByIssue(OfBizWorklogStore.java:125)
	at com.atlassian.jira.issue.worklog.DefaultWorklogManager.getByIssue(DefaultWorklogManager.java:94)  <+2> (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:134)
	at com.sun.proxy.$Proxy304.getByIssue(Unknown Source)  <+2> (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:154)
	at com.sun.proxy.$Proxy304.getByIssue(Unknown Source)
	at com.atlassian.jira.issue.worklog.WorklogManager$getByIssue$6.call(Unknown Source)
	at com.onresolve.jira.groovy.jql.ScriptFunctionSearchIndexer.addIndex(ScriptFunctionSearchIndexer.groovy:84)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.add(DefaultIssueDocumentFactory.java:102)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.addAll(DefaultIssueDocumentFactory.java:91)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:60)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:37)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:644)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:427)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:412)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:334)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:330)
	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:9)
	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:5)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:38)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:35)
	at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call(BoundedExecutor.java:85)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2015-07-22 10:47:51,235 IssueIndexer:thread-10 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 95% complete. Current index: Issue
2015-07-22 10:48:00,402 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex took: 675507ms. Indexer: DefaultIndexManager: paths: [/data/home/xxxxx/xxxxx/home/caches/indexes/comments,/data/home/xxxxx/xxxxx/home/caches/indexes/issues, /data/home/xxxxx/xxxxx/home/caches/indexes/changes, /data/home/xxxxx/xxxxx/home/caches/indexes/worklogs, //data/home/xxxxx/xxxxx/home/caches/indexes/plugins/atlassian-subversion-revisions]
2015-07-22 10:48:00,413 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 95% complete. Current index: SearchRequest
2015-07-22 10:48:01,867 SharedEntityIndexer:thread-5 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 96% complete. Current index: SearchRequest
2015-07-22 10:48:04,189 SharedEntityIndexer:thread-6 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 97% complete. Current index: SearchRequest
2015-07-22 10:48:06,463 SharedEntityIndexer:thread-10 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 98% complete. Current index: SearchRequest
2015-07-22 10:48:08,809 SharedEntityIndexer:thread-9 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 99% complete. Current index: SearchRequest
2015-07-22 10:48:09,415 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 99% complete. Current index: PortalPage
2015-07-22 10:48:11,004 SharedEntityIndexer:thread-5 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 100% complete. Current index: PortalPage
2015-07-22 10:48:11,006 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex took: 10590ms. Indexer: SharedEntityIndexManager: paths: []
2015-07-22 10:48:11,008 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 100% complete. Current index: 
2015-07-22 10:48:11,008 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx/secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 686097ms. Reindex run: 1
2015-07-22 10:48:11,009 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing finished

      Environment

      • JIRA version 6.4.8
      • JIRA Charting Plugin version 1.26.1

      Steps to Reproduce

      1. Upgrade from JIRA 5.2.11 to 6.4.8
      2. Install JIRA Charting Plugin version 1.26.1
      3. Perform a full re-index

      Expected Results

      1. Logs should not show any errors

      Actual Results

      The below exception is thrown in the atlassian-jira.log file:

      2015-07-22 10:47:43,844 IssueIndexer:thread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 94% complete. Current index: Issue
2015-07-22 10:47:44,522 IssueIndexer:thread-6 ERROR xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [core.entity.jdbc.SQLProcessor] !!! ABANDONED SQLProcessor DETECTED !!!
	This probably means that somebody forgot to close an EntityListIterator.
	Connection: DelegatingConnectionImpl[connectionPoolInfo=ConnectionPoolInfo{maxSize=30, minSize=2, initialSize=null, maxIdle=30, maxWait=60000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery='select 1', minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=null, validationQueryTimeout=null, defaultCatalog=null},sqlConnectionInterceptor=org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor@29d49e9a]
	SQL: select min( act.CREATED) from jiraaction act , jiraissue it where act.issueid = ?  and act.issueid = it.ID and act.actiontype = 'comment'  and it.REPORTER != act.AUTHOR
2015-07-22 10:47:44,524 IssueIndexer:thread-6 ERROR xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor] Unexpected exception from SQL connection interceptor callback: onConnectionReplaced([DelegatingConnectionImpl[connectionPoolInfo=ConnectionPoolInfo{maxSize=30, minSize=2, initialSize=null, maxIdle=30, maxWait=60000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery='select 1', minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=null, validationQueryTimeout=null, defaultCatalog=null},sqlConnectionInterceptor=org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor@29d49e9a], ConnectionPoolStateImpl[timeToBorrowNanos=0,borrowCount=2,connectionPoolInfo=ConnectionPoolInfo{maxSize=30, minSize=2, initialSize=null, maxIdle=30, maxWait=60000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery='select 1', minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=null, validationQueryTimeout=null, defaultCatalog=null}]])
java.lang.IllegalStateException: The OpTimer has been re-used.  end() can only be called once!
	at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:142)
	at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:99)
	at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:117)
	at com.atlassian.jira.ofbiz.InstrumentedSQLInterceptor.onConnectionReplaced(InstrumentedSQLInterceptor.java:39)
	at com.atlassian.jira.ofbiz.ChainedSQLInterceptor.onConnectionReplaced(ChainedSQLInterceptor.java:75)
	at org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor.onConnectionReplaced(SafeDelegatingSqlConnectionInterceptor.java:47)
	at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker$DelegatingConnectionImpl.close(ConnectionTracker.java:99)
	at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedConnection(ConnectionGuard.java:124)
	at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessor(ConnectionGuard.java:112)
	at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessors(ConnectionGuard.java:150)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:390)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:544)
	at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:960)
	at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:883)
	at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:730)
	at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:150)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:901)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:879)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findByAnd(DefaultOfBizDelegator.java:110)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findByAnd(WrappingOfBizDelegator.java:91)
	at com.atlassian.jira.issue.worklog.OfBizWorklogStore.getByIssue(OfBizWorklogStore.java:125)
	at com.atlassian.jira.issue.worklog.DefaultWorklogManager.getByIssue(DefaultWorklogManager.java:94)  <+2> (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:134)
	at com.sun.proxy.$Proxy304.getByIssue(Unknown Source)  <+2> (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:154)
	at com.sun.proxy.$Proxy304.getByIssue(Unknown Source)
	at com.atlassian.jira.issue.worklog.WorklogManager$getByIssue$6.call(Unknown Source)
	at com.onresolve.jira.groovy.jql.ScriptFunctionSearchIndexer.addIndex(ScriptFunctionSearchIndexer.groovy:84)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.add(DefaultIssueDocumentFactory.java:102)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.addAll(DefaultIssueDocumentFactory.java:91)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:60)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:37)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:644)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:427)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:412)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:334)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:330)
	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:9)
	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:5)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:38)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:35)
	at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call(BoundedExecutor.java:85)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2015-07-22 10:47:51,235 IssueIndexer:thread-10 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 95% complete. Current index: Issue
2015-07-22 10:48:00,402 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx/secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex took: 675507ms. Indexer: DefaultIndexManager: paths: [/data/home/xxxxx/xxxxx/home/caches/indexes/comments, /data/home/xxxxx/xxxxx/home/home/caches/indexes/issues, /data/home/xxxxx/xxxxx/home/caches/indexes/changes, /data/home/xxxxx/xxxxx/home/caches/indexes/worklogs, /data/home/xxxxx/xxxxx/home/caches/indexes/plugins/atlassian-subversion-revisions]
2015-07-22 10:48:00,413 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 95% complete. Current index: SearchRequest
2015-07-22 10:48:01,867 SharedEntityIndexer:thread-5 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 96% complete. Current index: SearchRequest
2015-07-22 10:48:04,189 SharedEntityIndexer:thread-6 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 97% complete. Current index: SearchRequest
2015-07-22 10:48:06,463 SharedEntityIndexer:thread-10 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 98% complete. Current index: SearchRequest
2015-07-22 10:48:08,809 SharedEntityIndexer:thread-9 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 99% complete. Current index: SearchRequest
2015-07-22 10:48:09,415 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 99% complete. Current index: PortalPage
2015-07-22 10:48:11,004 SharedEntityIndexer:thread-5 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 100% complete. Current index: PortalPage
2015-07-22 10:48:11,006 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex took: 10590ms. Indexer: SharedEntityIndexManager: paths: []
2015-07-22 10:48:11,008 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing is 100% complete. Current index: 
2015-07-22 10:48:11,008 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 686097ms. Reindex run: 1
2015-07-22 10:48:11,009 JiraTaskExectionThread-1 INFO xxxxxx 636x147x1 xhcmf7 10.x.xxx.xxx /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing finished

      Notes

      JCHART-462 is used to track the fix for this issue.

        1. index-fail.png
          index-fail.png
          19 kB

            [JRASERVER-44608] Error while re-indexing the DateOfFirstResponseCFType in the JIRA Charting Plugin

            Piotr Janik added a comment -

            Turning up in AT&ST scan results in Jira 9.12.x

            Piotr Janik added a comment - Turning up in AT&ST scan results in Jira 9.12.x

            Oswaldo Hernandez (Inactive) added a comment -

            Hi jasbris,

            The fix we prepared under this issue is definitely in version 1.33 of the charting plugin. If you are still seeing problems using that version of the plugin the best thing to do is to investigate the problem in the support case to find a root cause for the issue. I have let the support engineers know about this as well.

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - Hi jasbris , The fix we prepared under this issue is definitely in version 1.33 of the charting plugin. If you are still seeing problems using that version of the plugin the best thing to do is to investigate the problem in the support case to find a root cause for the issue. I have let the support engineers know about this as well. Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

            Jason Brison added a comment -

            ohernandez@atlassian.com, like cwhitten, we too see this in JIRA 6.4.11 with JIRA Charting Plugin v1.33.

            2015-11-12 12:15:51,989 IssueIndexer:thread-32 ERROR jbrison 681x619x1 1rmj23m 10.242.76.21,0:0:0:0:0:0:0:1 /secure/admin/jira/IndexReIndex.jspa [jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor] Unexpected exception from SQL connection interceptor callback: onConnectionReplaced([DelegatingConnectionImpl[connectionPoolInfo=ConnectionPoolInfo{maxSize=200, minSize=30, initialSize=null, maxIdle=20, maxWait=30000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery=null, minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=300, validationQueryTimeout=null, defaultCatalog=null},sqlConnectionInterceptor=org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor@7873638b], ConnectionPoolStateImpl[timeToBorrowNanos=0,borrowCount=7,connectionPoolInfo=ConnectionPoolInfo{maxSize=200, minSize=30, initialSize=null, maxIdle=20, maxWait=30000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery=null, minEvictableTimeMillis=null, timeBetweenEvictionRunsMillis=null, poolPreparedStatements=null, testOnBorrow=null, testOnReturn=null, testWhileIdle=null, maxOpenPreparedStatements=null, numTestsPerEvictionRun=null, removeAbandonedTimeout=300, validationQueryTimeout=null, defaultCatalog=null}]])
java.lang.IllegalStateException: The OpTimer has been re-used.  end() can only be called once!
        at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:142)
        at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:99)
        at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:117)
        at com.atlassian.jira.ofbiz.InstrumentedSQLInterceptor.onConnectionReplaced(InstrumentedSQLInterceptor.java:39)
        at com.atlassian.jira.ofbiz.ChainedSQLInterceptor.onConnectionReplaced(ChainedSQLInterceptor.java:75)
        at org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor.onConnectionReplaced(SafeDelegatingSqlConnectionInterceptor.java:47)
        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker$DelegatingConnectionImpl.close(ConnectionTracker.java:99)
        at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedConnection(ConnectionGuard.java:124)
        at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessor(ConnectionGuard.java:112)
        at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessors(ConnectionGuard.java:150)
        at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:390)
        at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:544)
        at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:960)
        at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:883)
        at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:730)
        at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:150)
        at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:901)
        at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:879)
        at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findByAnd(DefaultOfBizDelegator.java:110)
        at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findByAnd(WrappingOfBizDelegator.java:91)
        at com.atlassian.jira.issue.changehistory.DefaultChangeHistoryManager.getAllChangeItems(DefaultChangeHistoryManager.java:195)
        at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.apply(DefaultChangeHistoryRetriever.java:45)
        at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.apply(DefaultChangeHistoryRetriever.java:27)
        at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(DefaultIssueIndexer.java:655)
        at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(DefaultIssueIndexer.java:651)
        at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:642)
        at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:427)
        at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:412)
        at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:334)
        at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:330)
        at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:9)
        at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:5)
        at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:38)
        at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:35)
        at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call(BoundedExecutor.java:85)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

            Raised as JSP-253401

            Jason Brison added a comment - ohernandez@atlassian.com , like cwhitten , we too see this in JIRA 6.4.11 with JIRA Charting Plugin v1.33. 2015-11-12 12:15:51,989 IssueIndexer:thread-32 ERROR jbrison 681x619x1 1rmj23m 10.242.76.21,0:0:0:0:0:0:0:1 /secure/admin/jira/IndexReIndex.jspa [jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor] Unexpected exception from SQL connection interceptor callback: onConnectionReplaced([DelegatingConnectionImpl[connectionPoolInfo=ConnectionPoolInfo{maxSize=200, minSize=30, initialSize= null , maxIdle=20, maxWait=30000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery= null , minEvictableTimeMillis= null , timeBetweenEvictionRunsMillis= null , poolPreparedStatements= null , testOnBorrow= null , testOnReturn= null , testWhileIdle= null , maxOpenPreparedStatements= null , numTestsPerEvictionRun= null , removeAbandonedTimeout=300, validationQueryTimeout= null , defaultCatalog= null },sqlConnectionInterceptor=org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor@7873638b], ConnectionPoolStateImpl[timeToBorrowNanos=0,borrowCount=7,connectionPoolInfo=ConnectionPoolInfo{maxSize=200, minSize=30, initialSize= null , maxIdle=20, maxWait=30000, sleepTime=300000, lifeTime=600000, deadLockMaxWait=600000, deadLockRetryWait=10000, validationQuery= null , minEvictableTimeMillis= null , timeBetweenEvictionRunsMillis= null , poolPreparedStatements= null , testOnBorrow= null , testOnReturn= null , testWhileIdle= null , maxOpenPreparedStatements= null , numTestsPerEvictionRun= null , removeAbandonedTimeout=300, validationQueryTimeout= null , defaultCatalog= null }]]) java.lang.IllegalStateException: The OpTimer has been re-used. end() can only be called once! at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:142) at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:99) at com.atlassian.instrumentation.operations.SimpleOpTimer.end(SimpleOpTimer.java:117) at com.atlassian.jira.ofbiz.InstrumentedSQLInterceptor.onConnectionReplaced(InstrumentedSQLInterceptor.java:39) at com.atlassian.jira.ofbiz.ChainedSQLInterceptor.onConnectionReplaced(ChainedSQLInterceptor.java:75) at org.ofbiz.core.entity.jdbc.interceptors.connection.SafeDelegatingSqlConnectionInterceptor.onConnectionReplaced(SafeDelegatingSqlConnectionInterceptor.java:47) at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker$DelegatingConnectionImpl.close(ConnectionTracker.java:99) at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedConnection(ConnectionGuard.java:124) at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessor(ConnectionGuard.java:112) at org.ofbiz.core.entity.jdbc.ConnectionGuard.closeAbandonedProcessors(ConnectionGuard.java:150) at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:390) at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:544) at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:960) at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:883) at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:730) at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:150) at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:901) at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:879) at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findByAnd(DefaultOfBizDelegator.java:110) at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findByAnd(WrappingOfBizDelegator.java:91) at com.atlassian.jira.issue.changehistory.DefaultChangeHistoryManager.getAllChangeItems(DefaultChangeHistoryManager.java:195) at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.apply(DefaultChangeHistoryRetriever.java:45) at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.apply(DefaultChangeHistoryRetriever.java:27) at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(DefaultIssueIndexer.java:655) at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(DefaultIssueIndexer.java:651) at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:642) at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:427) at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:412) at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:334) at com.atlassian.jira.issue.index.DefaultIssueIndexer$3$1.get(DefaultIssueIndexer.java:330) at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:9) at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:5) at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:38) at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:35) at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call(BoundedExecutor.java:85) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Raised as JSP-253401

            Oswaldo Hernandez (Inactive) added a comment -

            This was addressed in JIRA Charting Plugin 1.33 and above cwhitten. If you are still seeing issues, can you please raise a support request at https://support.atlassian.com so we can investigate more about what is happening in your instance?

            Thanks in advance

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - This was addressed in JIRA Charting Plugin 1.33 and above cwhitten . If you are still seeing issues, can you please raise a support request at https://support.atlassian.com so we can investigate more about what is happening in your instance? Thanks in advance Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

            Chris Whitten [Comskil] added a comment -

            We also have JIRA 6.4.11 (server) and the latest version of the Charting plugin and still noticed the error:

            Chris Whitten [Comskil] added a comment - We also have JIRA 6.4.11 (server) and the latest version of the Charting plugin and still noticed the error:

            Oswaldo Hernandez (Inactive) added a comment -

            A JIRA 6.4.x compatible version of the add-on has been released with a fix to this problem, you can download this from https://marketplace.atlassian.com/download/plugins/com.atlassian.jira.ext.charting/version/2330

            JIRA Cloud will contain an updated plugin in the upcoming JIRA 7.0-OD-03 deployment.

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - A JIRA 6.4.x compatible version of the add-on has been released with a fix to this problem, you can download this from https://marketplace.atlassian.com/download/plugins/com.atlassian.jira.ext.charting/version/2330 JIRA Cloud will contain an updated plugin in the upcoming JIRA 7.0-OD-03 deployment. Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

            Dieter Greiner added a comment -

            msymons interesting to see that the Charting plugin crashes reindex in your case. In my case i saw the "error indexing issue" only during reindex that runs as final part of the upgrade process. Once the upgrade was done full reindex just produced errors in the log file but all issues were reindexed properly. But your case surely justifies to increase priority.

            It would be interesting to see if the workaround in Description helps in your case. As Atlassian support explained to me the code change just seems to lower the probability of a race condition within ofbiz code since closing the SQLprocessor runs faster that way. So likely the real culprit is not JIRA charting plugin though fixing the charting plugin is a reasonable workaround. (At least for me)
            Thus i strongly suggest to vote for JCHART-462 and maybe you'll make the same experience

            Dieter Greiner added a comment - msymons interesting to see that the Charting plugin crashes reindex in your case. In my case i saw the "error indexing issue" only during reindex that runs as final part of the upgrade process. Once the upgrade was done full reindex just produced errors in the log file but all issues were reindexed properly. But your case surely justifies to increase priority. It would be interesting to see if the workaround in Description helps in your case. As Atlassian support explained to me the code change just seems to lower the probability of a race condition within ofbiz code since closing the SQLprocessor runs faster that way. So likely the real culprit is not JIRA charting plugin though fixing the charting plugin is a reasonable workaround. (At least for me) Thus i strongly suggest to vote for JCHART-462 and maybe you'll make the same experience

            Mark Symons added a comment -

            Support (JSP-240743) have confirmed this issue as the cause of problems that I have been experiencing (resulting in many hours of overtime spent investigating) but the logged exception that I see is slightly different (wrt ofbiz)

            2015-07-30 17:25:37,891 IssueIndexer:thread-4 ERROR xxxx 1038x26582x1 1seoffq xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [core.entity.jdbc.SQLProcessor] !!! ABANDONED SQLProcessor DETECTED !!!
	This probably means that somebody forgot to close an EntityListIterator.
	Connection: org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker$DelegatingConnectionImpl@44738f9e
	SQL: null
2015-07-30 17:25:37,897 IssueIndexer:thread-4 WARN xxxx 1038x26582x1 1seoffq xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [jira.issue.index.DefaultIssueDocumentFactory] Error indexing issue xxx-1023: Dropping 'customfield_10231'
java.lang.IllegalStateException: The OpTimer has been re-used.  end() can only be called once!

            I believe that the following summary items in this issue are incorrect:

            • There are no visible errors on the JIRA UI.
            • The re-index finishes successfully bu there are errors in the logs

            ..as indexing can fail. UI displays nice big skull and crossbones with displayed exception:

            Log's referral number: 8d038900-8875-4fa4-9bad-f76fbd2b4323
Cause

Referer URL: Unknown

java.lang.NullPointerException

    java.lang.NullPointerException
    	at com.atlassian.jira.web.action.admin.index.IndexAdminImpl.doProgress(IndexAdminImpl.java:395)
    	at sun.reflect.GeneratedMethodAccessor1154.invoke(Unknown Source)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:497)
    	at webwork.util.InjectionUtils$DefaultInjectionImpl.invoke(InjectionUtils.java:70)
    	at webwork.util.InjectionUtils.invoke(InjectionUtils.java:56)
    	at webwork.action.ActionSupport.invokeCommand(ActionSupport.java:438)
    	at webwork.action.ActionSupport.execute(ActionSupport.java:157)
....

            with logged exception:

            2015-07-26 16:01:48,235 JiraTaskExectionThread-1 ERROR xxxx 946x2055x1 1867iq8 xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED.  Indexer: DefaultIndexManager: paths: [/data/jira/caches/indexes/comments, /data/jira/caches/indexes/issues, /data/jira/caches/indexes/changes, /data/jira/caches/indexes/worklogs]
com.atlassian.jira.index.IndexingFailureException: Indexing completed with 5 errors
	at com.atlassian.jira.index.AccumulatingResultBuilder$CompositeResult.await(AccumulatingResultBuilder.java:192)
	at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:1169)
	at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:1136)
	at com.atlassian.jira.issue.index.DefaultIndexManager.access$100(DefaultIndexManager.java:97)
	at com.atlassian.jira.issue.index.DefaultIndexManager$2.run(DefaultIndexManager.java:342)
	at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:382)
	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:337)
	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:287)  <+3> (NativeMethodAccessorImpl.java:62) (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83)
	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
	at com.sun.proxy.$Proxy11.reIndexAll(Unknown Source)
	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:75)
	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:55)
	at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:28)
	at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:59)
	at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:16)
	at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:24)
	at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:528)
	at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:254)
	at java.lang.Thread.run(Thread.java:745)
2015-07-26 16:01:48,237 JiraTaskExectionThread-1 INFO xxxx 946x2055x1 1867iq8 xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing finished

            (As per investigation log attached to JSP-240743, the culprit is definitely JIRA Charting Plugin).

            Additionally, Support Tools Plugin log-scan generates two alerts that link to CAC articles that were of zero help in fixing the problem:

            JIRA indexing failed after upgrade with Reindex ALL FAILED error
            JIRA Indexing Operations Display Error indexing issue Dropping Custom Field

            So, should this issue really be "minor"?

            Mark Symons added a comment - Support (JSP-240743) have confirmed this issue as the cause of problems that I have been experiencing (resulting in many hours of overtime spent investigating) but the logged exception that I see is slightly different (wrt ofbiz) 2015-07-30 17:25:37,891 IssueIndexer:thread-4 ERROR xxxx 1038x26582x1 1seoffq xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [core.entity.jdbc.SQLProcessor] !!! ABANDONED SQLProcessor DETECTED !!! This probably means that somebody forgot to close an EntityListIterator. Connection: org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker$DelegatingConnectionImpl@44738f9e SQL: null 2015-07-30 17:25:37,897 IssueIndexer:thread-4 WARN xxxx 1038x26582x1 1seoffq xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [jira.issue.index.DefaultIssueDocumentFactory] Error indexing issue xxx-1023: Dropping 'customfield_10231' java.lang.IllegalStateException: The OpTimer has been re-used. end() can only be called once! I believe that the following summary items in this issue are incorrect: There are no visible errors on the JIRA UI. The re-index finishes successfully bu there are errors in the logs ..as indexing can fail. UI displays nice big skull and crossbones with displayed exception: Log's referral number: 8d038900-8875-4fa4-9bad-f76fbd2b4323 Cause Referer URL: Unknown java.lang.NullPointerException java.lang.NullPointerException at com.atlassian.jira.web.action.admin.index.IndexAdminImpl.doProgress(IndexAdminImpl.java:395) at sun.reflect.GeneratedMethodAccessor1154.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at webwork.util.InjectionUtils$DefaultInjectionImpl.invoke(InjectionUtils.java:70) at webwork.util.InjectionUtils.invoke(InjectionUtils.java:56) at webwork.action.ActionSupport.invokeCommand(ActionSupport.java:438) at webwork.action.ActionSupport.execute(ActionSupport.java:157) .... with logged exception: 2015-07-26 16:01:48,235 JiraTaskExectionThread-1 ERROR xxxx 946x2055x1 1867iq8 xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED. Indexer: DefaultIndexManager: paths: [/data/jira/caches/indexes/comments, /data/jira/caches/indexes/issues, /data/jira/caches/indexes/changes, /data/jira/caches/indexes/worklogs] com.atlassian.jira.index.IndexingFailureException: Indexing completed with 5 errors at com.atlassian.jira.index.AccumulatingResultBuilder$CompositeResult.await(AccumulatingResultBuilder.java:192) at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:1169) at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:1136) at com.atlassian.jira.issue.index.DefaultIndexManager.access$100(DefaultIndexManager.java:97) at com.atlassian.jira.issue.index.DefaultIndexManager$2.run(DefaultIndexManager.java:342) at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:382) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:337) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:287) <+3> (NativeMethodAccessorImpl.java:62) (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83) at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28) at com.sun.proxy.$Proxy11.reIndexAll(Unknown Source) at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:75) at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:55) at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:28) at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:59) at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:16) at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:24) at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:528) at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:254) at java.lang.Thread.run(Thread.java:745) 2015-07-26 16:01:48,237 JiraTaskExectionThread-1 INFO xxxx 946x2055x1 1867iq8 xxx.xxx.xxx.xxx,127.0.0.1 /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing finished (As per investigation log attached to JSP-240743, the culprit is definitely JIRA Charting Plugin). Additionally, Support Tools Plugin log-scan generates two alerts that link to CAC articles that were of zero help in fixing the problem: JIRA indexing failed after upgrade with Reindex ALL FAILED error JIRA Indexing Operations Display Error indexing issue Dropping Custom Field So, should this issue really be "minor"?

              ohernandez@atlassian.com Oswaldo Hernandez (Inactive)
              astephen@atlassian.com Adrian Stephen
              Affected customers:
              5 This affects my team
              Watchers:
              19 Start watching this issue

                Created:
                Updated:
                Resolved: