Problem Definition

In the current implementation of DVCS, the errors recorded during sync does not follow a specific pattern and it's duplicated as well. It's hard to identify if the same error was hit multiple times during the repo sync or performing a specific action.

We'd record same exception multiple times with multiple log patterns making it difficult and almost do some guess work:

• Exception would sometimes have Repo ID but would not say which action (REST call during sync) failed
• Exception would sometimes have Repo name in the URL but would not have the Repo ID or would not say which action failed
• Exception would only have the action (REST API) which failed but would not say which repo ID or repo name

For Example:

• Here it doesn't have any information on which repository hit this error or performing which action. Even it's hard for jira admin to know that [c.a.j.p.d.sync.impl.DefaultSynchronizer] is actually from com.atlassian.jira.plugins.dvcs class since there's no URL reference:

  2022-11-25 02:31:22,968+0000 Caesium-1-4 ERROR ServiceRunner     [c.a.j.p.d.sync.impl.DefaultSynchronizer] java.io.IOException: Bad Gateway (502)
  java.lang.RuntimeException: java.io.IOException: Bad Gateway (502)
  	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.IssueCommentPayloadEventProcessor.getPullRequestByHtmlUrl(IssueCommentPayloadEventProcessor.java:89)
  	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.IssueCommentPayloadEventProcessor.process(IssueCommentPayloadEventProcessor.java:52)
  	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.GitHubEventProcessorAggregatorImpl.process(GitHubEventProcessorAggregatorImpl.java:64)
  	at com.atlassian.jira.plugins.dvcs.service.GitHubEventServiceImpl$SyncTransactionCallback.doInTransaction(GitHubEventServiceImpl.java:223)
  	at com.atlassian.jira.plugins.dvcs.service.GitHubEventServiceImpl$SyncTransactionCallback.doInTransaction(GitHubEventServiceImpl.java:177)
  

• Here it's easier to say this is for the repository ID 18792 and during a softsync - however, it still doesn't say which action got this error:

  2022-11-25 03:47:59,377+0000 http-nio-8080-exec-37 url: /rest/bitbucket/1.0/repository/18792/softsync; user: xxxxx ERROR xxxxxx 227x4502705x2 kxlh0i x.x.x.x,y.y.y.y,z.z.z.z /rest/bitbucket/1.0/repository/18792/softsync [c.a.j.p.d.sync.impl.DefaultSynchronizer] java.io.IOException: Bad Gateway (502)
  java.lang.RuntimeException: java.io.IOException: Bad Gateway (502)
  	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.IssueCommentPayloadEventProcessor.getPullRequestByHtmlUrl(IssueCommentPayloadEventProcessor.java:89)
  	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.IssueCommentPayloadEventProcessor.process(IssueCommentPayloadEventProcessor.java:52)
  	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.GitHubEventProcessorAggregatorImpl.process(GitHubEventProcessorAggregatorImpl.java:64)
  	at com.atlassian.jira.plugins.dvcs.service.GitHubEventServiceImpl$SyncTransactionCallback.doInTransaction(GitHubEventServiceImpl.java:223)
  	at com.atlassian.jira.plugins.dvcs.service.GitHubEventServiceImpl$SyncTransactionCallback.doInTransaction(GitHubEventServiceImpl.java:177)
  

• Here we can see which action failed with the error - however, here we don't see which repository ID/name affected:

  2022-11-25 02:31:22,967+0000 Caesium-1-4 WARN ServiceRunner     [c.a.j.p.d.spi.github.GithubClientWithTimeout] Failed to execute request [GET /repositories/123456/pulls?state=closed&per_page=100&page=2] errors: 
      Bad Gateway (502)
  java.io.IOException: Bad Gateway (502)
  	at org.eclipse.egit.github.core.client.GitHubClient.createException(GitHubClient.java:564)
  	at org.eclipse.egit.github.core.client.GitHubClient.get(GitHubClient.java:740)
  

Suggested Solution

• Create a unique Org+repo hash key when an Org/Repo is added to DVCS accounts
• Make the logging per repo sync consistent so that the errors during the sync can be extracted with this hash
  • For example if I want to see every error hit during a particular repo sync, I can just use this hash and extract the entire sync history from the logs

    Workaround

    Cross reference errors you think is related to the repo sync and validate from the database to make sure the error we see in the logs is the same error recorded in the database:

    select * from "AO_E8B6CC_REPOSITORY_MAPPING" RM
    JOIN "AO_E8B6CC_SYNC_AUDIT_LOG" AL
    ON RM."ID" = AL."REPO_ID"
    WHERE AL."TOTAL_ERRORS" !=0 AND RM."NAME" = '<repository name>';