Loading...

Type: Bug
Resolution: Fixed
Priority: Low
Fix Version/s: 10.2.0
Affects Version/s: 8.20.19
Component/s: Indexing
Labels:

Introduced in Version:
8.2
Support reference count:
19
Symptom Severity:
Severity 3 - Minor
UIS:
15
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary

When an instance has many filters, the SearchRequest re-index phase will take a longer time.

If the time goes above the <pool-remove-abandoned-timeout> time from the dbconfig.xml (300s by default) then the database connection from the JiraTaskExecutionThread-# thread is considered abandoned.

Usually, this results in only a warning like this, and no practical problems:

2023-03-30 01:03:03,739+0000 pool-29-thread-1 WARN admin 62x37x1 14ju9qw 192.168.0.4 /secure/admin/jira/IndexReIndex!reindex.jspa [atlassian-diagnostics] 2023-03-30T01:03:03.680Z Component 'Database' alerted 'Database connection leaked' (details: {"activeConnections":2,"idleConnections":19,"maxConnections":100,"connectionAcquiredTimestampInMillis":1680138123564}, trigger: {"pluginKey": "not-detected"})

If the instance is using many DB connections or the parameter <pool-remove-abandoned-on-maintenance> is set to true, the connection could be evicted, resulting in the error below and failure in the re-index process.

2023-03-30 01:03:10,674+0000 JiraTaskExecutionThread-1 ERROR admin 62x37x1 14ju9qw 172.29.226.177,172.50.0.4 /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindexing encountered an error for one of the indexers. Reindex all continues. Error for indexer: SharedEntityIndexManager: paths: []
org.ofbiz.core.util.GeneralRuntimeException: Error getting the next result (This ResultSet is closed.)

This causes some filters to be missing from the index and can affect boards, dashboards and queues.

This is reproducible on Data Center: yes

Steps to Reproduce

Create about 100k-300k filters. For example, using a loop like this:

for i in {1..300000}; do echo " Filter $i"; curl -D- -u username:pasword -X POST  -H "Content-Type: application/json" --data "{ \"name\": \"All Open Bugs $i\", \"description\": \"Lists all open bugs\", \"jql\": \"type = Bug and resolution is empty\", \"favourite\": true, \"editable\": true}" https://myjira.com/rest/api/2/filter ; done

Reduce the <pool-remove-abandoned-timeout> to 60 seconds (or less if needed) in the dbconfig.xml
Add the parameter <pool-remove-abandoned-on-maintenance>true</pool-remove-abandoned-on-maintenance>
Restart Jira
When running the reindex, if the SearchRequest part takes more than the time set to <pool-remove-abandoned-timeout>, the reindex should fail.

Expected Results

The re-index operation finishes successfully.

Actual Results

We see that after the time set in <pool-remove-abandoned-timeout> (60 seconds in this case), the "Database connection leaked" message is logged.
After a few seconds, the connection is evicted and the error happens, resulting in the re-index failure.

2023-03-30 01:02:03,502+0000 JiraTaskExecutionThread-1 INFO admin 62x37x1 14ju9qw 172.29.226.177,172.50.0.4 /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: SearchRequest
...
2023-03-30 01:03:03,739+0000 pool-29-thread-1 WARN admin 62x37x1 14ju9qw 172.29.226.177,172.50.0.4 /secure/admin/jira/IndexReIndex!reindex.jspa [atlassian-diagnostics] 2023-03-30T01:03:03.680Z Component 'Database' alerted 'Database connection leaked' (details: {"activeConnections":2,"idleConnections":19,"maxConnections":100,"connectionAcquiredTimestampInMillis":1680138123564}, trigger: {"pluginKey": "not-detected"})
...
2023-03-30 01:03:10,674+0000 JiraTaskExecutionThread-1 ERROR admin 62x37x1 14ju9qw 172.29.226.177,172.50.0.4 /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindexing encountered an error for one of the indexers. Reindex all continues. Error for indexer: SharedEntityIndexManager: paths: []
org.ofbiz.core.util.GeneralRuntimeException: Error getting the next result (This ResultSet is closed.)
	at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:272)
	at com.atlassian.jira.ofbiz.DefaultOfBizListIterator.next(DefaultOfBizListIterator.java:100)
	at com.atlassian.jira.ofbiz.WrappingOfBizListIterator.next(WrappingOfBizListIterator.java:68)
	at com.atlassian.jira.ofbiz.DatabaseIterator.pullNext(DatabaseIterator.java:57)
	at com.atlassian.jira.ofbiz.DatabaseIterator.populateNextIfNull(DatabaseIterator.java:66)
	at com.atlassian.jira.ofbiz.DatabaseIterator.hasNext(DatabaseIterator.java:37)
	at com.atlassian.jira.util.collect.CollectionUtil.foreach(CollectionUtil.java:38)
	at com.atlassian.jira.util.collect.CloseableIterator$Functions.foreach(CloseableIterator.java:50)
	at com.atlassian.jira.ofbiz.DatabaseIterable.foreach(DatabaseIterable.java:39)
	at com.atlassian.jira.util.collect.TransformingEnclosedIterable.foreach(TransformingEnclosedIterable.java:26)
	at com.atlassian.jira.sharing.index.DefaultSharedEntityIndexManager.reIndex(DefaultSharedEntityIndexManager.java:124)
	at com.atlassian.jira.sharing.index.DefaultSharedEntityIndexManager.reIndexAll(DefaultSharedEntityIndexManager.java:93)
	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:70)
	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:51)
	at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:27)
	at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
	at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:18)
	at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
	at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:533)
	at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: This ResultSet is closed.
	at org.postgresql.jdbc.PgResultSet.checkClosed(PgResultSet.java:2932)
	at org.postgresql.jdbc.PgResultSet.next(PgResultSet.java:1951)
	at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:191)
	at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:191)
	at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:266)

Workaround

There are a few approaches to avoid this problem:

This should solve the issue in most cases - remove <pool-remove-abandoned-on-maintenance> from dbconfig.xml and restart Jira.
Increase the <pool-remove-abandoned-timeout> time in dbconfig.xml and restart Jira;
Improve the re-index performance. Here are some ideas:
- Increase the number of threads by setting the jira.index.sharedentity.threads parameter in jira-config.properties. The default value is 10.
- Improve the database resources;
- Start the re-index from a node that has lower latency to the database

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(2 mentioned in)

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Forms

Activity

People

Dates