High Description
The symptom is that all Users (also sysadmin) can't log into JIRA. The system just responds with a failed authentication attempt and the log prints out a stack trace like this:
com.atlassian.jira.util.dbc.Assertions$NullArgumentException: user should not be null!
at com.atlassian.jira.util.dbc.Assertions.notNull(Assertions.java:26)
at com.atlassian.jira.security.login.LoginManagerImpl.authorise(LoginManagerImpl.java:135)
at com.atlassian.jira.security.JiraRoleMapper.canLogin(JiraRoleMapper.java:46)
at com.atlassian.seraph.auth.DefaultAuthenticator.isAuthorised(DefaultAuthenticator.java:229)
at com.atlassian.seraph.auth.DefaultAuthenticator.authoriseUserAndEstablishSession(DefaultAuthenticator.java:197)
at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:102)
at com.atlassian.crowd.integration.seraph.v22.CrowdAuthenticator.login(CrowdAuthenticator.java:133)
at com.atlassian.seraph.filter.PasswordBasedLoginFilter.runAuthentication(PasswordBasedLoginFilter.java:127)
at com.atlassian.seraph.filter.PasswordBasedLoginFilter.login(PasswordBasedLoginFilter.java:72)
at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:130)
at com.atlassian.jira.web.filters.JiraLoginFilter.doFilter(JiraLoginFilter.java:70)
Judging from the code, the actual authentication with Crowd already succeeded (com.atlassian.seraph:atlassian-seraph:2.5.0/com.atlassian.seraph.auth.DefaultAuthenticator.login(HttpServletRequest, HttpServletResponse, String, String, boolean)/line 94:111):
final boolean authenticated = authenticate(principal, password); if (dbg) { log.debug(METHOD + "'" + userName + "' has " + (authenticated ? "been" : "not been") + " authenticated"); } if (authenticated) { final Principal user = getUser(userName); if (authoriseUserAndEstablishSession(httpServletRequest, httpServletResponse, user)) { if (setRememberMeCookie && httpServletResponse != null) { getRememberMeService().addRememberMeCookie(httpServletRequest, httpServletResponse, userName); } return true; } AUTHORISATION_FAILED.stampRequestResponse(httpServletRequest, httpServletResponse); }
The authenticate call in the first line is the direct authentication with Crowd. The authoriseUserAndEstablishSession(HttpServletRequest, HttpServletResponse, Principal) call corresponds with line 102 of the stack trace above. The problem is that the getUser(String) call before returns null for the user. This call leads to com.atlassian.jira.security.login.JiraSeraphAuthenticator.getUser(String) which just uses com.atlassian.crowd.embedded.core.CrowdServiceImpl.getUser(String) in order to retrieve the user. This is its code:
Application application = getApplication();
if (application == null)
{
return null;
}
try
{
return applicationService.findUserByName(application, name);
}
catch (com.atlassian.crowd.exception.UserNotFoundException e)
{
return null;
}
getApplication() is ultimately retrieving com.atlassian.crowd.dao.application.ApplicationDAO (cwd_application table) with name crowd-embedded. This tuple is contained in the db:
jira=> select * from cwd_application; id | application_name | lower_application_name | created_date | updated_date | active | description | application_type | credential ----+------------------+------------------------+----------------------------+----------------------------+--------+-------------+------------------+------------ 1 | crowd-embedded | crowd-embedded | 2011-05-08 14:09:05.607-07 | 2011-05-08 14:09:05.607-07 | 1 | | CROWD | X (1 row)
The com.atlassian.crowd.manager.application.ApplicationServiceGeneric.findUserByName(Application, String) code looks as following:
for (final Directory directory : getActiveDirectories(application)) { try { return directoryManager.findUserByName(directory.getId(), name); } catch (final UserNotFoundException e) { // user not in directory, keep cycling } catch (final DirectoryNotFoundException e) { // directory not found throw new ConcurrentModificationException("Directory mapping was removed while iterating through directories: " + e.getMessage()); } catch (final OperationFailedException e) { // directory has some massive error, keep cycling logger.error(e.getMessage(), e); } } // could not find user in any of the directories throw new UserNotFoundException(name);
It iterates over the active directories and tries to find the user.
jira=> select * from cwd_directory;
id | directory_name | lower_directory_name | created_date | updated_date | active | description | impl_class | lower_impl_class | directory_type | directory_position
----+-------------------------+-------------------------+----------------------------+----------------------------+--------+---------------------------------+----------------------------------------------------+----------------------------------------------------+----------------+--------------------
2 | Remote Crowd Directory | remote crowd directory | 2011-05-08 14:09:06.142-07 | 2011-05-08 23:50:57.531-07 | 1 | Remote crowd directory | com.atlassian.crowd.directory.RemoteCrowdDirectory | com.atlassian.crowd.directory.remotecrowddirectory | CROWD | 0
1 | JIRA Internal Directory | jira internal directory | 2011-05-08 14:09:06.166-07 | 2011-05-08 23:51:08.015-07 | 0 | JIRA default internal directory | com.atlassian.crowd.directory.InternalDirectory | com.atlassian.crowd.directory.internaldirectory | INTERNAL | 1
(2 rows)
The select above shows that there's only one active directory called 'Remote Crowd Directory' which points to the Crowd instance:
jira=> select * from cwd_directory_attribute where directory_id=2;
directory_id | attribute_name | attribute_value
--------------+---------------------------------------------------------+----------------------------------------
2 | com.atlassian.crowd.directory.sync.lastdurationms | 70
2 | com.atlassian.crowd.directory.sync.laststartsynctime | 1304923857460
2 | com.atlassian.crowd.directory.sync.currentstartsynctime |
2 | useNestedGroups | false
2 | com.atlassian.crowd.directory.sync.issynchronising | false
2 | application.name | jira
2 | application.password | foobar
2 | crowd.server.url | http://<customerinstance>.jira.com/crowd/
(8 rows)
The actual directory implementation will lead to com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(String) which will just ask com.atlassian.crowd.embedded.ofbiz.OfBizUserDao.findByName(long, String) in the end. The implementation seems to be a bridge for the legacy OfBizUser code. It only asks a ConcurrentMap<DirectoryEntityKey, OfBizUser> userCache first with the username and then with the lowercased username:
// Try with the case we have been given. // This is an optimisation that should work in most JIRA instances where group and user names are really all lower case. OfBizUser user = userCache.get(DirectoryEntityKey.getKeyPreserveCase(directoryId, userName)); if (user != null) { return user; } user = userCache.get(DirectoryEntityKey.getKey(directoryId, userName)); if (user == null) { // Because the SPI says we should do this. throw new UserNotFoundException(userName); } return user;
I hooked up a debugger to a customer instance and it turns out that userCache is empty. Judging from the implementation, there has to be an initial call to fill the cache. There are three public methods which would add users, com.atlassian.crowd.embedded.ofbiz.OfBizUserDao.flushCache() which only has com.atlassian.crowd.embedded.ofbiz.OfBizUserDao.onEvent(XMLRestoreFinishedEvent) as a client (import), com.atlassian.crowd.embedded.ofbiz.OfBizUserDao.add(User, PasswordCredential) and com.atlassian.crowd.embedded.ofbiz.OfBizUserDao.addAll(Set<UserTemplateWithCredentialAndAttributes>). One indirect client of addAll is com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJob.execute(JobExecutionContext) which I suspect to be responsible for the (initial) syncing. That job either didn't run or had problems.
In order to workaround the job not syncing, we can trigger the synchronisation manually on /plugins/servlet/embedded-crowd/directories/list. If the directories are not synced, cwd_user, cwd_group and cwd_membership will be empty for that directory. So in order to get into the system, you have to manually create the sysadmin (nopass is not used since we only authenticate against Crowd, see code above):
insert into cwd_user (id, directory_id, user_name, lower_user_name, active, created_date, updated_date, first_name, lower_first_name, last_name, lower_last_name, display_name, lower_display_name, email_address, lower_email_address, credential) values (1, 2, 'sysadmin', 'sysadmin', 1, now(), now(), 'System', 'system', 'Administrator', 'administrator', 'System Administrator', 'system administrator', 'noreply@atlassian.com', 'noreply@atlassian.com', 'nopass'); insert into cwd_group (id, group_name, lower_group_name, active, local, created_date, updated_date, group_type, directory_id) values (1, 'system-administrators', 'system-administrators', 1, 0, now(), now(), 'GROUP', 2); insert into cwd_membership (id, parent_id, child_id, membership_type, parent_name, lower_parent_name, child_name, lower_child_name, directory_id) values (1, 1, 1, 'GROUP_USER', 'system-administrators', 'system-administrators', 'sysadmin', 'sysadmin', 2);
