Details
-
Suggestion
-
Resolution: Unresolved
-
None
-
None
-
None
-
6
-
15
-
Description
NOTE: This suggestion is for JIRA Server. Using JIRA Cloud? See the corresponding suggestion.
Problem Definition
Currently, if the user that is tied to a token in oauthsptoken does not exist in cwd_user table - this makes one of the Scheduler job; CompatibilityPluginScheduler.JobId.Service Provider Session Remover fails with the following error:
2016-07-27 09:02:38,879 Caesium-1-3 ERROR ServiceRunner [c.a.scheduler.core.JobLauncher] Scheduled job with ID 'CompatibilityPluginScheduler.JobId.Service Provider Session Remover' failed com.atlassian.oauth.serviceprovider.InvalidTokenException: Token 'DeEdwSKAgEMamU39X50H4uMX7yx6Uwqg' is an access token, but has no user associated with it at com.atlassian.jira.oauth.serviceprovider.OfBizServiceProviderTokenStore.createTokenFromGV(OfBizServiceProviderTokenStore.java:322) at com.atlassian.jira.oauth.serviceprovider.OfBizServiceProviderTokenStore.removeExpiredTokens(OfBizServiceProviderTokenStore.java:250) at com.atlassian.jira.oauth.serviceprovider.OfBizServiceProviderTokenStore.removeExpiredTokensAndNotify(OfBizServiceProviderTokenStore.java:238) at com.atlassian.jira.oauth.serviceprovider.CachingServiceProviderTokenStore.removeExpiredTokensAndNotify(CachingServiceProviderTokenStore.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ... 2 filtered at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) at com.sun.proxy.$Proxy1194.removeExpiredTokensAndNotify(Unknown Source) at com.atlassian.oauth.serviceprovider.internal.DelegatingTokenStoreImpl.removeExpiredTokensAndNotify(DelegatingTokenStoreImpl.java:48) at com.atlassian.oauth.serviceprovider.internal.ExpiredSessionRemover.execute(ExpiredSessionRemover.java:25) at com.atlassian.scheduler.compat.clustered.ClusteredJobRunner.runJob(ClusteredJobRunner.java:54) at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153) at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118) at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:401) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:396) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:349) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:255) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:252) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34) at java.lang.Thread.run(Thread.java:745)
That means the table oauthsptoken will keep growing in size because the job that suppose to purge the data in that table is being triggered and fails due to 1 or more invalid ACCESS token(s). Due to this, the records in the table is kept without any purging and also it creates some invalid REQUEST tokens in the table, for example:
| 490159 | 2016-08-31 15:54:04 | 2diaN9RRfbp9QslzjyjJWzh5NuNj4nIr | TVYUpHnFDdUvPeHVR0JMTkCC7QC9n3Ys | REQUEST | Confluence:xxxxxxxxx | NULL | 600000 | NONE | http://<baseURL>/confluence/plugins/servlet/gadgets/oauth-callback | NULL | V_1_0_A | NULL | NULL | NULL | NULL | | 490158 | 2016-08-31 15:53:48 | 8oCvjMa5Vk7LceN3FWxSYvZNf0u7zmpA | SHJbAmOWMmbXOGu1ywBNpLLQ4UkA4V9C | REQUEST | Confluence:xxxxxxxxx | NULL | 600000 | NONE | http://<baseURL>/confluence/plugins/servlet/gadgets/oauth-callback | NULL | V_1_0_A | NULL | NULL | NULL | NULL | | 490157 | 2016-08-31 15:53:32 | eHdogIdSLpq6CtTsCb1xt5LPDc7eIElW | cc3dbtHiauvlN5tJxP19pjEIb68pNm94 | REQUEST | Confluence:xxxxxxxxx | NULL | 600000 | NONE | http://<baseURL>/confluence/plugins/servlet/gadgets/oauth-callback | NULL | V_1_0_A | NULL | NULL | NULL | NULL | | 490156 | 2016-08-31 15:53:16 | w4NKljZKmKekJezqwj89Aao0neVf1yNl | 9suIWzuSM9B4oTUCW7LLZs3fdEmEs9Eh | REQUEST | Confluence:xxxxxxxxx | NULL | 600000 | NONE | http://<baseURL>/confluence/plugins/servlet/gadgets/oauth-callback | NULL | V_1_0_A | NULL | NULL | NULL | NULL |
Suggested Solution
Remove those tokens from oauthsptoken table if the user does not exist in cwd_user table - why do we need to keep tokens for a user that does not exist in JIRA anymore?
Workaround
Always take a DB backup before you are removing any data from any DB table.
Option 1
- Find the invalid tokens by running the following SQL and remove them from oauthsptoken table.:
select * from oauthsptoken where token_type = 'ACCESS' and username not in (select lower_user_name from cwd_user);
This SQL is not perfect - you will need to cross check those users from the result in app_user table to make sure those users got renamed before.
- As an alternative, this SQL will be more accurate but this is not tested:
select * from oauthsptoken where token_type = 'ACCESS' and username not in (select user_key from app_user)
Option 2
- Wait for the job to be triggered and remove the token that the job is complaining about.
- From the error above, we can see that token DeEdwSKAgEMamU39X50H4uMX7yx6Uwqg is the one that is failing. Remove the token from the oauthsptoken table by using this SQL:
delete from oauthsptoken where token='DeEdwSKAgEMamU39X50H4uMX7yx6Uwqg';
Replace the token with the error that you are getting. - Beside the log file, you will be able to check the error at Administration > System > Scheduler details page, find for CompatibilityPluginScheduler.JobRunnerKey.Service Provider Session Remover, click on Show more. The error message is this page will give you the token that you will need to remove.
- This job is scheduled to run in 8 hours interval, which means if you have few invalid token, then you will need to wait for the Next run - to get the next token that you will need to remove to DB.
Attachments
Issue Links
- relates to
-
JRASERVER-44422 Scheduler Administration JiraPluginScheduler shows a java.lang.NullPointerException exception
-
- Gathering Impact
-
-
- Closed