Status: Closed (View Workflow)
7.13.0, 7.6.9, 8.3.1, 7.13.12, 8.13.0
Severity 2 - Major
Please check this KB to check some details on how this bug was fixed and what other problems related to user-login were fixed: https://confluence.atlassian.com/jirakb/user-login-jira-stats-logs-1108675859.html
Data inconsistency result in users not able to login. There is a number of related cases when for all of them the root cause isn’t found, for most of them DB manipulation/refreshing cache/re-adding directory and syncing it solved the problem.
This results in users unable to login, but flows/actions that could result in this state are plenty of. Mainly those are user login, directory Synchronisation.
For some customers that works only temporary so the workaround is done over and over again.
Inconsistency exists in DB between cwd_membership and cwd_user tables.
The most probable reason of it is cache corruption. It was confirmed in 1 case.
There is a KnowledgeBase article for that, number of cases with similar symptoms and the same workaround steps.
The biggest problem with pointing out the root cause is that problem show off with the delay(like a couple of months later), when there are no available logs providing information how data inconsistency was created (due to failed CRUD operation, or cache corruption -> that's why there are different workaround - manual DB update / new user-directory + resync & cache refresh)
- Problem is not yet reproducible
- On affected/corrupted instances users are not able to login
An active user is able to login. If a problem occurs, sync AD with Jira solves the problem.
-> There is no data inconsistency
The below exception is thrown in the Atlassian-jira.log file:
We have reports from a group of customers that suggests that problem was caused by staging/testing Jira instance sending the cache updates to the production cluster. In other words, production data in cwd_membership cache got poisoned by data from another environment.
- This usually happens when production data backup snapshot is restored to lower environments without cleaning the cluster nodes information.
- To prevent this, please remove all production nodes from lower environments after restore, please check Remove abandoned or offline nodes in JIRA Data Center
- We have a KB explaining problem in detail and covering diagnostic steps: Inconsistency in group membership and user status on one or multiple nodes in Jira Datacenter
- Problem is lagerly mitigated in Jira 8.10+ with introduction of node clean-up job , see
- Refresh caches by full Jira restart (all nodes in case of DC)
- Full restart in Data Center is required because nodes with a corrupted cache will replicate it to other starting nodes, this means shutting down all nodes, validating the service is stopped and starting them back up, a rolling restart will not resolve this issue.
- Add new directory, sync it, remove the 1st one
- Create a new user directory with exact same settings as Directory ID: 10000 (name: Active Directory server
- Synchronize with this new directory
- Move this new directory to top position (the first directory to be checked)
- Test user access
- Remove old directory(id:10000)
- Remove blocking(inconsistent data) user from cwd_membership and consequently update cwd_user:
We should be able to see if the DB is affected by running:
Following updates will remove any inconsistent data. Always have a backup of your database before doing any changes to it
A cold restart is needed to rebuilt the cache with the correct data.