Highest Description
According to https://jira.atlassian.com/browse/BAM-13886, this issue should have been fixed. But according to the logs and the effects we've observed - failed builds - the solution doesn't work reliable. Let me tell a story ....
A particular agent is idle ..
2014-01-21 03:09:29,654 INFO [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [BuildAgentControllerImpl] Agent 'agent-16-14.buildeng.atlassian.com' ready to take build from queue...
A plugin is installed on the server
2014-01-21 03:15:36,846 INFO [pool-16-thread-4] [PluginSettingsAuditLogService] Tue Jan 21 03:15:36 UTC 2014 anonymous: Installed add-on auto-restart (com.atlassian.bamboo.plugins.auto-restart version 1.0-beta-3-SNAPSHOT)
The update is propagated to the agent
2014-01-21 03:14:53,764 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.DisableRemotePluginEvent[source=null] 2014-01-21 03:14:53,767 INFO [Timer-1] [DefaultStopNeededPollingService] Stop needed polling service stopped 2014-01-21 03:15:07,370 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.UpgradeRemotePluginEvent[source=null] 2014-01-21 03:15:07,371 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Syncing [/opt/bamboo-agent/plugins/system-provided] with https://jira-bamboo.internal.atlassian.com/agentServer/bundled-plugins.zip?finger print=4931149357869138675 2014-01-21 03:15:07,429 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] 9 file(s) already available in [/opt/bamboo-agent/plugins/system-provided] 2014-01-21 03:15:07,457 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Syncing [/opt/bamboo-agent/plugins/user-installed] with https://jira-bamboo.internal.atlassian.com/agentServer/plugins.zip?fingerprint=4931149357869138675 2014-01-21 03:15:07,708 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] 14 file(s) already available in [/opt/bamboo-agent/plugins/user-installed] 2014-01-21 03:15:11,138 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [BootstrapUtils] 2 MB transferred 2014-01-21 03:15:12,741 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [BootstrapUtils] 4 MB transferred 2014-01-21 03:15:15,849 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [BootstrapUtils] 6 MB transferred 2014-01-21 03:15:16,971 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Removing '/opt/bamboo-agent/plugins/user-installed/plugin_4397322126637599494_auto-restart-1.0-beta-3-SNAPSHOT.jar' from Agent classpath 2014-01-21 03:15:16,972 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Creating '/opt/bamboo-agent/plugins/user-installed/plugin_1611035189066388876_bamboo-plan-templates-1.0-SNAPSHOT.jar' 2014-01-21 03:15:16,999 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [BootstrapUtils] 2 MB transferred 2014-01-21 03:15:17,029 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [BootstrapUtils] 4 MB transferred 2014-01-21 03:15:17,055 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [BootstrapUtils] 6 MB transferred 2014-01-21 03:15:21,159 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteAgentPluginUpgradedEventListener] Found 1 new plugins 2014-01-21 03:15:21,163 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] 2014-01-21 03:15:31,183 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginEvent[source=null] 2014-01-21 03:15:31,184 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteAgentPluginEnabledEventListener] Enabling plugin 'com.atlassian.bamboo.plugins.auto-restart' 2014-01-21 03:15:31,197 INFO [Spring executor 8] [StopNeededChecker] Stop file set as "/opt/bamboo-agent/bamboo-agent-restart", will stop agent if file is found. 2014-01-21 03:15:31,198 INFO [Spring executor 8] [DefaultStopNeededPollingService] Stop needed polling service started
Odd - the log says it is removing the plugin and yet it is enabled.
Anyway, time flies by and another plugin is installed on the server
2014-01-21 03:24:02,394 INFO [pool-16-thread-2] [PluginSettingsAuditLogService] Tue Jan 21 03:24:02 UTC 2014 anonymous: Installed add-on broker-url-discovery (com.atlassian.bamboo.buildeng.brokertask.brokerurldiscovery version 1.0-alpha-10-SNAPSHOT)
And the update is propagated to the agent
2014-01-21 03:22:45,240 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.DisableRemotePluginEvent[source=null] 2014-01-21 03:22:53,992 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.UpgradeRemotePluginEvent[source=null] 2014-01-21 03:22:53,993 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Syncing [/opt/bamboo-agent/plugins/system-provided] with https://jira-bamboo.internal.atlassian.com/agentServer/bundled-plugins.zip?fingerprint=4931149357869138675 2014-01-21 03:22:54,000 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] 9 file(s) already available in [/opt/bamboo-agent/plugins/system-provided] 2014-01-21 03:22:54,025 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Syncing [/opt/bamboo-agent/plugins/user-installed] with https://jira-bamboo.internal.atlassian.com/agentServer/plugins.zip?fingerprint=4931149357869138675 2014-01-21 03:22:54,075 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] 14 file(s) already available in [/opt/bamboo-agent/plugins/user-installed] 2014-01-21 03:22:59,264 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Removing '/opt/bamboo-agent/plugins/user-installed/plugin_5696379602079403040_brokerurldiscovery-1.0-alpha-10-SNAPSHOT.jar' from Agent classpath 2014-01-21 03:22:59,264 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [ClasspathBuilder] Creating '/opt/bamboo-agent/plugins/user-installed/plugin_4711894070437594952_auto-restart-1.0-beta-3-SNAPSHOT.jar' 2014-01-21 03:22:59,785 INFO [Timer-1] [DefaultStopNeededPollingService] Stop needed polling service stopped 2014-01-21 03:22:59,819 INFO [Spring executor 11] [StopNeededChecker] Stop file set as "/opt/bamboo-agent/bamboo-agent-restart", will stop agent if file is found. 2014-01-21 03:22:59,820 INFO [Spring executor 11] [DefaultStopNeededPollingService] Stop needed polling service started 2014-01-21 03:23:00,812 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteAgentPluginUpgradedEventListener] Found 1 new plugins 2014-01-21 03:23:05,073 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] 2014-01-21 03:23:09,918 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] 2014-01-21 03:23:15,483 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] 2014-01-21 03:23:38,245 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] 2014-01-21 03:23:38,246 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] 2014-01-21 03:23:38,246 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] 2014-01-21 03:23:38,248 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null]
Now the auto-restart plugin is installed (but apparently not enabled); and even worse, the brokerurldiscovery plugin is gone.
For whatever reason, the agent starts a build which is actually requiring the brokerurldiscovery and sadly, it fails.
2014-01-21 03:24:15,652 INFO [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [BuildAgentControllerImpl] A1. JIRA Master Tier 1 - CI Functional Tests - TF-272-clear-comment-when-canceling-wiki-preview - Func 11 #2 (MASTERONE-CIFUNCHAL75-FUNC11-2) taken from queue. ... 2014-01-21 03:24:41,953 ERROR [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [TaskExecutorImpl] Could not execute task no Plugin with key 'com.atlassian.bamboo.buildeng.brokertask.brokerurldiscovery:brokerKey' is installed. ... 2014-01-21 03:24:42,158 INFO [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [RemoteResultProcessor] Build A1. JIRA Master Tier 1 - CI Functional Tests - TF-272-clear-comment-when-canceling-wiki-preview - Func 11 #2 (MASTERONE-CIFUNCHAL75-FUNC11-2) completed on Bamboo agent. Sending results to server. 2014-01-21 03:24:42,194 INFO [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [BuildAgentControllerImpl] Agent 'agent-16-14.buildeng.atlassian.com' ready to take build from queue...
and another one ...
2014-01-21 03:33:10,555 INFO [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [BuildAgentControllerImpl] A2. JIRA Master Tier 2 - CI Webdriver - CAS-571-fixing-flakey-test - WebDriver Batch 07 #8 (MASTERTWO-FIREFOXHAL57-WEBDRIVER07-8) taken from queue. ... 2014-01-21 03:33:54,074 ERROR [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [TaskExecutorImpl] Could not execute task no Plugin with key 'com.atlassian.bamboo.buildeng.brokertask.brokerurldiscovery:brokerKey' is installed. ... 2014-01-21 03:34:03,913 INFO [3-BAM::agent-16-14.buildeng.atlassian.com::Agent:pool-6-thread-1] [BuildAgentControllerImpl] Agent 'agent-16-14.buildeng.atlassian.com' ready to take build from queue...
After 20 minutes, the brokerurldiscorvery plugin is enabled. There is no message in the log when it was installed.
2014-01-21 03:46:01,147 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] Rebroadcasting com.atlassian.bamboo.plugin.events.EnableRemotePluginModuleEvent[source=null] ... 2014-01-21 03:46:01,174 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteAgentPluginEnabledEventListener] Enabling plugin 'com.atlassian.bamboo.buildeng.brokertask.brokerurldiscovery'
Attachments
Issue Links
- was cloned as
-
BDEV-4270 Loading...