Loading...

Details

Type: Bug
Resolution: Fixed
Priority: Medium
Fix Version/s: 8.16.0, 8.13.6
Affects Version/s: 8.12.0, 8.13.1, 8.13.2
Component/s: Data Center - Index, Data Center - Node replication
Labels:

Introduced in Version:
8.12
Support reference count:
12
Symptom Severity:
Severity 2 - Major
UIS:
58
Bug Fix Policy:
View Atlassian Server bug fix policy

Description

Issue Summary

nodereindex replication fails for single issues with 1000 or more worklog entries.

Steps to Reproduce

Create a sample Jira issue

Add 1001 worklog entries.

 #!/bin/bash 
 COUNTER=1001
 until [  $COUNTER -lt 10 ]; do
     curl -u admin:sphere -H "Content-Type: application/json" -X POST "http://jira:8080/rest/api/2/issue/TEST-1/worklog" -d '{"comment":"workin over here","started":"2018-02-21T00:00:00.000+0000","timeSpentSeconds":12000}'
     let COUNTER-=1
 done

Perform an operation that calls the indexer on this test issue.

Expected Results

The change is successfully replicated.

Actual Results

The below exception is thrown in the atlassian-jira.log and the change is not replicated to worklog index.

2021-01-12 08:03:02,121+0000 NodeReindexServiceThread:thread-0 ERROR      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Error re-indexing node changes
java.lang.RuntimeException: Failed to update index for ([ReplicatedIndexOperation{indexTime=2021-01-12 08:02:39.327831, id=47337, nodeId='9eaf0c1e601143d294f323a79d5d97c1', affectedIdToVersion={10000=16470}, operation=UPDATE_WITH_RELATED, affectedIndex=ISSUE, entityType=NONE, backupFilename=''}, ReplicatedIndexOperation{indexTime=2021-01-12 08:02:39.348218, id=47338, nodeId='9eaf0c1e601143d294f323a79d5d97c1', affectedIdToVersion={1010000=4}, operation=UPDATE, affectedIndex=COMMENT, entityType=NONE, backupFilename=''}, ReplicatedIndexOperation{indexTime=2021-01-12 08:02:39.377009, id=47339, nodeId='9eaf0c1e601143d294f323a79d5d97c1', affectedIdToVersion={1015823=63, 1015822=63, 1015821=63, 1015820=63, 1015819=63, 1015818=63, 1015817=63, 1015816=63, 1015815=63, 

<snipped> 

1015793=63, 1015792=63}, operation=UPDATE, affectedIndex=WORKLOG, entityType=NONE, backupFilename=''}])
    at com.atlassian.jira.index.ha.DefaultNodeReindexService.applyIndexOperations(DefaultNodeReindexService.java:347)
    at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:307)
    at com.atlassian.jira.util.thread.JiraThreadLocalUtils.lambda$wrap$1(JiraThreadLocalUtils.java:156)
    at com.atlassian.jira.util.concurrent.ScheduledExecutorServiceUtils$1.run(ScheduledExecutorServiceUtils.java:43)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: The size of set with ids must less than 1000
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
    at com.atlassian.jira.issue.worklog.DefaultWorklogManager.getWorklogsForIds(DefaultWorklogManager.java:176)
    at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateWorklogsIndex(DefaultNodeReindexService.java:832)
    at com.atlassian.jira.index.ha.DefaultNodeReindexService.processWorklogIndexingOperations(DefaultNodeReindexService.java:490)
    at com.atlassian.jira.index.ha.ZDUAwareNodeReindexService.processWorklogIndexingOperations(ZDUAwareNodeReindexService.java:100)
    at com.atlassian.jira.index.ha.DefaultNodeReindexService.applyIndexOperations(DefaultNodeReindexService.java:344)
    ... 10 more

nodereindex will further retry the replication which will naturally fail again due to the 1000 count limitation:

2021-01-12 08:24:30,957+0000 NodeReindexServiceThread:thread-0 INFO      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Retrying indexing of FailedReplicatedIndexOperation{tries=4, maxRetries=4, attemptTime=2021-01-12T08:24:28.402, operationsCount=3}
2021-01-12 08:24:31,366+0000 NodeReindexServiceThread:thread-0 ERROR      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Failed re-attempting to replicate index operations for (FailedReplicatedIndexOperation{tries=4, maxRetries=4, attemptTime=2021-01-12T08:24:28.402, operationsCount=3})
java.lang.IllegalArgumentException: The size of set with ids must less than 1000
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
	at com.atlassian.jira.issue.worklog.DefaultWorklogManager.getWorklogsForIds(DefaultWorklogManager.java:176)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateWorklogsIndex(DefaultNodeReindexService.java:832)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.processWorklogIndexingOperations(DefaultNodeReindexService.java:490)
	at com.atlassian.jira.index.ha.ZDUAwareNodeReindexService.processWorklogIndexingOperations(ZDUAwareNodeReindexService.java:100)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:464)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.retryPreviouslyFailedOperations(DefaultNodeReindexService.java:359)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:310)
	at com.atlassian.jira.util.thread.JiraThreadLocalUtils.lambda$wrap$1(JiraThreadLocalUtils.java:156)
	at com.atlassian.jira.util.concurrent.ScheduledExecutorServiceUtils$1.run(ScheduledExecutorServiceUtils.java:43)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-01-12 08:24:31,367+0000 NodeReindexServiceThread:thread-0 ERROR      [c.a.j.index.ha.FailedReplicationOperationService] Reached threshold of maximum retries for replication operation FailedReplicatedIndexOperation{tries=5, maxRetries=4,

Workaround

Execute the query below to find our issues that have more than 1000 worklog entries:

select issueid, count(id) from worklog group by issueid having count(id) > 1000 order by (count(id)) desc;

Clone these affected issues and either delete or archive the original ones.

Attachments

Issue Links

is related to

JRASERVER-73874 Reindexing issues with large number of comments, worklogs, history overloads indexers

Closed

JRASERVER-71976 DBR replication Fails on issues with high count of worklog entries

Closed

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

relates to: DBR-277 Loading...

(5 mentioned in, 1 relates to)

nodereindex replication fails for single issues with 1000 or more worklog entries