Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-21417

Jira access token getting expired after 2 hours while connecting to Gitlab

XMLWordPrintable

      Description

      This ticket is a continuation of JSWSERVER-21389. Our team thought the problem was resolved, but later it appeared we ran in troubles with concurrency, that's why that one (JSWSERVER-21417) was created. We have decided to create second ticket because root cause is slightly different and it affects next released versions than it's mentioned in JSWSERVER-21389.

      Issue Summary

      This is reproducible on Data Center: (yes) 

      Steps to Reproduce

      • Install GitLab 15.0.
      • Connect Jira to GitLab 
      • Wait 2 hours
      • Try to perform full sync, this will fail intermittently

      Expected Results

      The token refresh should take place in the background and sync should work

      Actual Results

      The token expires and cannot be refreshed. Looks to be issue with threads in order to overwrite the new token.Below are the error logs

      2022-07-22 16:50:25,612+0000 ajp-nio-127.0.0.1-8009-exec-397 url: /rest/bitbucket/1.0/repository/705/fullsync; user: xxxxxxxxx TRACE xxxxxxxxx 1010x56004x1 1c71cnj 10.60.128.173 /rest/bitbucket/1.0/repository/705/fullsync [c.a.o.c.lib.token.DefaultTokenService] Got OAuth Provider response: [400], [{"error":"invalid_grant","error_description":"The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."}
    ]
2022-07-22 16:50:25,614+0000 ajp-nio-127.0.0.1-8009-exec-397 url: /rest/bitbucket/1.0/repository/705/fullsync; user: xxxxxxxxx DEBUG xxxxxxxxx 1010x56004x1 1c71cnj 10.60.128.173 /rest/bitbucket/1.0/repository/705/fullsync [c.a.j.p.d.gitlab.impl.OAuthTokenProviderImpl] Failed to get access token

...

2022-07-22 16:51:46,322+0000 Caesium-1-2 DEBUG ServiceRunner     [c.a.j.p.d.gitlab.impl.AbstractGitLabRestClientImpl] Received response [GET https://gitlab.assist.zone/api/v4/users?search=xxxxxxx&per_page=100 returned a response status of 200 OK]
2022-07-22 16:51:46,322+0000 Caesium-1-2 INFO ServiceRunner     [c.a.j.p.dvcs.service.RepositoryServiceImpl] Could not load user [xxxxxx, xxxxxxx]
com.atlassian.jira.plugins.dvcs.exception.SourceControlException: java.lang.NullPointerException
      2022-07-25 13:08:06,585+0000 DVCSConnector.MessageExecutor:thread-48 DEBUG xxxxxxxxx 768x98182x1 10c6dyi 10.60.128.173 /rest/bitbucket/1.0/repository/846/fullsync [c.a.j.p.dvcs.event.ThreadEvents] There is no active ThreadEventsCaptor. Dropping event: DevSummaryChangedEvent{date=Mon Jul 25 13:08:06 UTC 2022, issueKeys=[DEP-512], repositoryId=846, dvcsType='gitlabselfmanaged'}

2022-07-25 13:08:06,590+0000 DVCSConnector.MessageExecutor:thread-48 DEBUG xxxxxxxxxx 768x98182x1 10c6dyi 10.60.128.173 /rest/bitbucket/1.0/repository/846/fullsync [c.a.j.p.d.service.message.MessageConsumer] Could not acquire token for consumer: 'com.atlassian.jira.plugins.dvcs.sync.GitLabSynchronizeChangesetMessageConsumer@23319f60' with id 'com.atlassian.jira.plugins.dvcs.spi.gitlab.message.GitLabSynchronizeChangesetMessage'. There are already '2' threads processing the messages.

      mysql> select * from AO_723324_CLIENT_TOKEN \G
*************************** 1. row ***************************
            ACCESS_TOKEN: 3bb0d420b0f586f9f7b43aec2263572b46d70c1c7233451c329f1d250c75d049
 ACCESS_TOKEN_EXPIRATION: 1658509102379
               CONFIG_ID: aef8a204-aa84-4085-babe-42cc67d5b37a
                      ID: 216a24ff-d0e9-4f7e-87d4-619e12d233dd
          LAST_REFRESHED: NULL
     LAST_STATUS_UPDATED: 1658753719903
           REFRESH_COUNT: 0
           REFRESH_TOKEN: acca34b4145b496b99c362715a2331f08a58fa4957d8601a7b1600570f8322f3
REFRESH_TOKEN_EXPIRATION: 1661094301379
                  STATUS: UNRECOVERABLE

      Workaround

      Workaround is not very helpful because it consist in recreating GitLab integration after every 2 hours.

          Form Name

            [JSWSERVER-21417] Jira access token getting expired after 2 hours while connecting to Gitlab

            Rusi Popov added a comment -

            The problem is analyzed with JIRA 8.20.11 and GitLab 15.4.3 ee in https://jira.atlassian.com/browse/JSWSERVER-21243?focusedCommentId=3189438&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-3189438

            Rusi Popov added a comment - The problem is analyzed with JIRA 8.20.11 and GitLab 15.4.3 ee in https://jira.atlassian.com/browse/JSWSERVER-21243?focusedCommentId=3189438&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-3189438

            Rohit Sharma added a comment -

            We have JIRA v9.1.0 (Upgraded to get the DVCS connector working with Gitlab) as the gitlab was forcing token expiration and JIRA DVCS had to provide a fix to use the refresh token. Without this the repo sync was stopping after two hours for the repositories which were populated.

            Once we upgraded to Gitlab ee 14.2.2 (latest) its broken again now and this time its even worse and it cannot even populate the list of repos to sync.

            Rohit Sharma added a comment - We have JIRA v9.1.0 (Upgraded to get the DVCS connector working with Gitlab) as the gitlab was forcing token expiration and JIRA DVCS had to provide a fix to use the refresh token. Without this the repo sync was stopping after two hours for the repositories which were populated. Once we upgraded to Gitlab ee 14.2.2 (latest) its broken again now and this time its even worse and it cannot even populate the list of repos to sync.

            Suzanne Cooley added a comment -

            Jarrod, thank you for your response!

            I am new to using Jira and this was the first issue I had encountered, so when I found this ticket, I was unaware how separate the development for Cloud and Server/Data Centre are.

            I will log a ticket for this issue with Jira Cloud.

            Thanks you!

            Suzanne Cooley added a comment - Jarrod, thank you for your response! I am new to using Jira and this was the first issue I had encountered, so when I found this ticket, I was unaware how separate the development for Cloud and Server/Data Centre are. I will log a ticket for this issue with Jira Cloud. Thanks you!

            Jarrod Price (Inactive) added a comment -

            Hi all,

            This issue was specifically for Jira Server and Data Centre, this same issue can not be used for any Cloud development as the development and support teams are completely separate.

            I have searched for an issue for Jira Cloud and I cannot find anything at all with this same problem description, therefore there is no official public response from our developers.

            I would urge you to please log relevant support tickets for your own organizations and link your Jira Cloud instances, then ask the assigned cloud support engineer to raise an issue with the matching problem as this issue.

            This is the best way for us to track affected customers with this same problem and to get an official response from our developers regarding whether or not they will provide a fix.

            Kind Regards,
            Jarrod Price
            Jira On-Prem Support, EMEA
            ATLASSIAN

            Jarrod Price (Inactive) added a comment - Hi all, This issue was specifically for Jira Server and Data Centre, this same issue can not be used for any Cloud development as the development and support teams are completely separate. I have searched for an issue for Jira Cloud and I cannot find anything at all with this same problem description, therefore there is no official public response from our developers. I would urge you to please log relevant support tickets for your own organizations and link your Jira Cloud instances, then ask the assigned cloud support engineer to raise an issue with the matching problem as this issue. This is the best way for us to track affected customers with this same problem and to get an official response from our developers regarding whether or not they will provide a fix. Kind Regards, Jarrod Price Jira On-Prem Support, EMEA ATLASSIAN

            BostjanR added a comment -

            ...is it maybe because "GitLab.com for Jira Cloud" plugin is preferred over DVCS for connecting Cloud Jira to GitLab.com?

            BostjanR added a comment - ...is it maybe because "GitLab.com for Jira Cloud" plugin is preferred over DVCS for connecting Cloud Jira to GitLab.com?

            yangyingming added a comment - - edited

            The official staff told me they were not going to fix the problem in Jira Cloud

             

            yangyingming added a comment - - edited The official staff told me they were not going to fix the problem in Jira Cloud  

            Suzanne Cooley added a comment -

            Hello, I am using Jira Cloud and have the same issue with Gitlab. Is this fix only for Jira server? Is there a corresponding ticket for the issue with Jira Cloud?

            Suzanne Cooley added a comment - Hello, I am using Jira Cloud and have the same issue with Gitlab. Is this fix only for Jira server? Is there a corresponding ticket for the issue with Jira Cloud?

            Jarrod Price (Inactive) added a comment -

            Hi fd0ec3dff0d8,

            At the moment the only workaround is to sync repos one at a time using a fresh token each time, it's quite painful I must admit especially if you have dozens or hundreds of repos syncing with Jira, and if a single repo takes more than 2 hours to sync, it's not possible at all.

            You could also consider downgrading Gitlab to the previous version below 15, I understand the inconvenience but I'm only able to offer suggestions for the time being.

            Our devs are still working on it and I'm myself waiting for a response regarding v9.1.

            Kind Regards,
            Jarrod Price
            Jira On-Prem Support, EMEA
            ATLASSIAN

            Jarrod Price (Inactive) added a comment - Hi fd0ec3dff0d8 , At the moment the only workaround is to sync repos one at a time using a fresh token each time, it's quite painful I must admit especially if you have dozens or hundreds of repos syncing with Jira, and if a single repo takes more than 2 hours to sync, it's not possible at all. You could also consider downgrading Gitlab to the previous version below 15, I understand the inconvenience but I'm only able to offer suggestions for the time being. Our devs are still working on it and I'm myself waiting for a response regarding v9.1. Kind Regards, Jarrod Price Jira On-Prem Support, EMEA ATLASSIAN

            Caterina Trainito added a comment - - edited

            Is the issue fixed in 8.20.11? On this ticket it is listed in the Affects Versions, while on JSWSERVER-21389 in the Fix Versions.

            Caterina Trainito added a comment - - edited Is the issue fixed in 8.20.11? On this ticket it is listed in the Affects Versions, while on JSWSERVER-21389 in the Fix Versions.

            Eshwar Palem added a comment -

            Hi Jarrod Price,

            We are running on Jira 8.20.5 and I don't think our team will be ready for an upgrade to another minor version 8.20.11 as our last upgrade was a few months back and we cannot upgrade it again just for this fix. Is there any other workaround?

            Eshwar Palem added a comment - Hi Jarrod Price, We are running on Jira 8.20.5 and I don't think our team will be ready for an upgrade to another minor version 8.20.11 as our last upgrade was a few months back and we cannot upgrade it again just for this fix. Is there any other workaround?

              1c017edd2c81 awojtasik
              3814e4a56ec7 Aniket Ranjan
              Affected customers:
              14 This affects my team
              Watchers:
              37 Start watching this issue

                Created:
                Updated:
                Resolved: