-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
High
-
None
-
Affects Version/s: 9.1.0
-
Component/s: Security Monitoring and Alerts
-
1
-
Severity 2 - Major
-
9
Issue Summary
Atlassian Security Monitoring and Alerts is bundled with Bitbucket Data Center and monitors user and admin changes to raise security alerts. When a user's details are changed, the plugin performs its security-alert processing synchronously on the same thread that published the change event. This includes checking whether the affected user is a system administrator or administrator before raising a security alert. That permission check can be expensive, holding up the event thread and blocking the process that triggered the event.
This affects user-detail changes, username changes, and password changes.
Steps to Reproduce
- Have the admin user-change alerts enabled in Atlassian Security Monitoring and Alerts.
- In an environment where resolving admin/permission status is slow (for example, a large remote user directory with nested groups), change a user's details, username, or password so that the corresponding event is published.
- Capture a thread dump while the operation is in progress and observe that the thread publishing the event is blocked inside the plugin's permission-check processing rather than returning immediately.
Expected Results
The plugin's security-alert processing should not hold up the thread that published the event while performing permission checks. This processing should be offloaded onto a separate thread, for example via an executor service.
Actual Results
The event thread is blocked for an extended period inside the plugin's alert processing while it resolves permission status for an alert notification. The relevant frames from a captured thread dump are (JDBC driver frames omitted below, as the exact frames depend on the database in use):
java.lang.Thread.State: RUNNABLE ... (blocked on a database call) at com.atlassian.crowd.dao.user.UserDAOHibernate.search(UserDAOHibernate.java:426) at com.atlassian.stash.internal.crowd.HibernateUserDao.search(HibernateUserDao.java:290) ... at com.atlassian.crowd.embedded.core.CrowdServiceImpl.search(CrowdServiceImpl.java:157) at com.atlassian.stash.internal.crowd.RiotPolice.getGroupsByUser(RiotPolice.java:543) at com.atlassian.stash.internal.user.DefaultUserService.getGroupsByUser(DefaultUserService.java:269) ... at com.atlassian.stash.internal.permission.GrantedPermissionVoterProvider.loadGroupMembership(GrantedPermissionVoterProvider.java:440) at com.atlassian.cache.memory.MemoryCacheManager$2.load(MemoryCacheManager.java:194) ... at com.atlassian.stash.internal.permission.GrantedPermissionVoterProvider$NormalUserPermissionVoterCalculator.calculate(GrantedPermissionVoterProvider.java:485) at com.atlassian.stash.internal.permission.GrantedPermissionVoterProvider$RecalculatingPermissionVoter.vote(GrantedPermissionVoterProvider.java:578) at com.atlassian.stash.internal.permission.CompositePermissionVoter.vote(CompositePermissionVoter.java:69) at com.atlassian.stash.internal.user.PermissionServiceImpl.hasGlobalPermission(PermissionServiceImpl.java:108) ... at com.atlassian.stash.internal.sal.user.DefaultUserManager.hasGlobalPermission(DefaultUserManager.java:203) at com.atlassian.stash.internal.sal.user.DefaultUserManager.isAdmin(DefaultUserManager.java:136) ... at com.atlassian.lighthouse.service.user.BitbucketUserDetailsFinder.fromUsername(BitbucketUserDetailsFinder.kt:83) at com.atlassian.lighthouse.notification.template.TemplateParametersService.toDisplayNameByUsername(TemplateParametersService.kt:157) at com.atlassian.lighthouse.notification.template.TemplateParametersService.access$toDisplayNameByUsername(TemplateParametersService.kt:47) at com.atlassian.lighthouse.notification.template.TemplateParametersService$getRenderedExtraAttributes$3$1.invoke(TemplateParametersService.kt:114) at com.atlassian.lighthouse.notification.template.TemplateParametersService.applyTransformFunctionForAttributes(TemplateParametersService.kt:244) at com.atlassian.lighthouse.notification.template.TemplateParametersService.getRenderedExtraAttributes(TemplateParametersService.kt:114) at com.atlassian.lighthouse.notification.template.TemplateParametersService.getAlertSpecificRenderingParameters(TemplateParametersService.kt:102) at com.atlassian.lighthouse.notification.template.MailTemplateFactory.getAlertSpecificRenderingParameters(MailTemplateFactory.kt:38) at com.atlassian.lighthouse.notification.template.MailTemplateFactory.createTemplate(MailTemplateFactory.kt:30) at com.atlassian.lighthouse.notification.NotificationService.sendEmailNotification(NotificationService.kt:33) at com.atlassian.lighthouse.security.service.AlertService.trySendNotification(AlertService.kt:50) at com.atlassian.lighthouse.security.service.AlertService.persistAlertAndSendNotification(AlertService.kt:34) at com.atlassian.lighthouse.service.group.AdminChangeHandler.handleUserChangeEvent(AdminChangeHandler.kt:67) at com.atlassian.lighthouse.eventlisteners.UserDetailsChangeListener$onUserEdited$1.invoke(UserDetailsChangeListener.kt:29) at com.atlassian.lighthouse.eventlisteners.AbstractBaseAtlassianEventListener.handleWhenAlertIsEnabled(AbstractBaseAtlassianEventListener.kt:32) at com.atlassian.lighthouse.eventlisteners.UserDetailsChangeListener.onUserEdited(UserDetailsChangeListener.kt:28) ... at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:103) at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.internalPublish(TransactionAwareEventPublisher.java:175) at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.publish(TransactionAwareEventPublisher.java:108) at com.atlassian.stash.internal.crowd.DelegatingMultiEventPublisher.publishAll(DelegatingMultiEventPublisher.java:28) at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.publishEvent(DbCachingRemoteChangeOperations.java:1055) at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.updateUsers(DbCachingRemoteChangeOperations.java:294) ... at com.atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations.addOrUpdateCachedUsers(DirectoryCacheImplUsingChangeOperations.java:58) at com.atlassian.crowd.directory.ldap.cache.RemoteDirectoryCacheRefresher.synchroniseAllUsers(RemoteDirectoryCacheRefresher.java:94) at com.atlassian.crowd.directory.synchronisation.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:45) at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseAll(EventTokenChangedCacheRefresher.java:62) at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1119) at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:95)
The blocked thread is performing an external-directory (LDAP) cache synchronisation, which publishes a user-changed event as part of that synchronisation. The plugin's event listener handles this event synchronously and, while resolving permission status for the alert, ends up performing a blocking database call. As a result, the directory synchronisation itself is held up for the duration of that call.
Workaround
Temporarily disable the affected alert types by adding the following entry to bitbucket.properties, then restarting Bitbucket. See Monitor security threats for more information on alert IDs and this setting:
plugin.lighthouse.disabled.alert.types=admin-user-details-modified,admin-username-modified,admin-user-password-modified
- If the property is already configured, append these alert IDs to the existing comma-separated list rather than replacing its current values.
- The disabled alert types are loaded when the plugin starts up, so a restart is required for the change to take effect.
This disables alerts for admin user-detail changes, username changes, and password changes. The underlying security changes are still recorded in Bitbucket's audit logs. This setting does not disable auditing. Other alert types remain enabled.
Once a fix is available and deployed, remove the added alert IDs from the property and restart Bitbucket to re-enable the alerts. Preserve any unrelated alert IDs that were already configured.