Summary

      After the weekly release of 30-Aug-2015 some Custom AMIs aren't able to start properly. They will remain pending and will eventually be shut down.

      Environment

      • Bamboo Cloud
      • Custom AMI
      • Using JDK6

      Steps to Reproduce

      1. Use a custom AMI
      2. Make sure the JAVA_HOME is pointed to JDK6
      3. Save the custom AMI and try to start up the image from Bamboo

      Expected Results

      EC2 instance and agent start up normally.

      Actual Results

      The Agent fails to start up and throws the below exception in the agent.out file:

      2015-08-31 08:15:34,843 FATAL [Thread-1] [AgentBootstrap] Exiting due to fatal exception.
      java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:28)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'buildAgentController' defined in class path resource [applicationContextRemoteAgent.xml]: Cannot resolve reference to bean 'agentQueueAccessor' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentQueueAccessor' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Cannot resolve reference to bean 'connectionFactory' while setting bean property 'connectionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Cannot create inner bean 'com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory#2cf5d030' of type [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory#2cf5d030' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory]: Constructor threw exception; nested exception is java.lang.IncompatibleClassChangeError: Class java.net.Socket does not implement the requested interface java.io.Closeable
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:336)
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
      	at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:646)
      	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:140)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1114)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1017)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
      	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
      	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
      	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
      	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
      	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703)
      	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
      	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
      	at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
      	at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105)
      	at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:227)
      	at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:70)
      	at com.atlassian.bamboo.agent.elastic.client.ElasticAgent.start(ElasticAgent.java:31)
      	... 6 more
      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentQueueAccessor' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Cannot resolve reference to bean 'connectionFactory' while setting bean property 'connectionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Cannot create inner bean 'com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory#2cf5d030' of type [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory#2cf5d030' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory]: Constructor threw exception; nested exception is java.lang.IncompatibleClassChangeError: Class java.net.Socket does not implement the requested interface java.io.Closeable
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:336)
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1456)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1197)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
      	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
      	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
      	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
      	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
      	... 25 more
      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Cannot create inner bean 'com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory#2cf5d030' of type [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory#2cf5d030' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory]: Constructor threw exception; nested exception is java.lang.IncompatibleClassChangeError: Class java.net.Socket does not implement the requested interface java.io.Closeable
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:290)
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122)
      	at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:646)
      	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:140)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1114)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1017)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
      	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
      	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
      	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
      	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
      	... 35 more
      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory#2cf5d030' defined in class path resource [applicationContextRemoteAgentJMS.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory]: Constructor threw exception; nested exception is java.lang.IncompatibleClassChangeError: Class java.net.Socket does not implement the requested interface java.io.Closeable
      	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1114)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1017)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:276)
      	... 47 more
      Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory]: Constructor threw exception; nested exception is java.lang.IncompatibleClassChangeError: Class java.net.Socket does not implement the requested interface java.io.Closeable
      	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:164)
      	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:125)
      	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:270)
      	... 52 more
      Caused by: java.lang.IncompatibleClassChangeError: Class java.net.Socket does not implement the requested interface java.io.Closeable
      	at com.google.common.io.Closeables.close(Closeables.java:80)
      	at com.google.common.io.Closeables.closeQuietly(Closeables.java:99)
      	at com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory.ping(BambooActiveMQConnectionFactory.java:191)
      	at com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory.validateUri(BambooActiveMQConnectionFactory.java:145)
      	at com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory.setBrokerURL(BambooActiveMQConnectionFactory.java:63)
      	at org.apache.activemq.ActiveMQConnectionFactory.<init>(ActiveMQConnectionFactory.java:142)
      	at org.apache.activemq.ActiveMQConnectionFactory.<init>(ActiveMQConnectionFactory.java:138)
      	at org.apache.activemq.ActiveMQSslConnectionFactory.<init>(ActiveMQSslConnectionFactory.java:75)
      	at com.atlassian.bamboo.v2.build.agent.BambooActiveMQConnectionFactory.<init>(BambooActiveMQConnectionFactory.java:53)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:148)
      	... 54 more
      

      Notes

      This will affect older custom images which were created when Java 6 was still the default JDK installed in Atlassian instances.

      Workaround

      Install JDK8u45 and make sure that JAVA_HOME is properly set system-wide to point at JDK8u45. This will allow the Bamboo Agent to start up again normally.

      Downloads of previous versions of Java can be found at the Oracle Java Archive page.

            [BAM-16291] Elastic agent stuck in Pending state with Java6

            Mike Egar added a comment -

            Hi Simon,
            Could you please let us know which 3rd party libraries are incompatible with java versions greater than 8u45 and when they will support later versions?

            Thanks,
            Mike

            Mike Egar added a comment - Hi Simon, Could you please let us know which 3rd party libraries are incompatible with java versions greater than 8u45 and when they will support later versions? Thanks, Mike

            Hi everyone,

            We apologise for the lack of communication for the end of support of Java 6. We announced the deprecation of support for Java 6 (November 2014) but did not communicate the date of the end of support for JDK6 prior the release of Bamboo Cloud on Monday, September 31st.

            With the end of support for JDK 7 (deprecated in March 2015) also approaching in the upcoming release of Bamboo 5.10 OD5 planned for Sept 28th, we highly recommend upgrading to JDK 8u45 as soon as possible to avoid any issues.

            Some recommendations:

            • Use JDK 8u45. The latest JDKs are incompatible with some 3rd party libraries we're using.
            • Try to match the layout and scripts of our stock images as closely as possible. This will make it easier for us to provide help if anything goes wrong.
            • Choose Oracle if you have the choice between Oracle and OpenJDK flavor of JDK.

            We are not able to make Bamboo Cloud compatible with JDK6 due to the nature of the changes recently introduced in the code and custom AMIs need to be upgraded. Please contact our Support Team if you need assistance with the upgrade.

            Best regards,

            Sten Pittet
            Bamboo Product Manager

            Sten Pittet (Inactive) added a comment - Hi everyone, We apologise for the lack of communication for the end of support of Java 6. We announced the deprecation of support for Java 6 ( November 2014 ) but did not communicate the date of the end of support for JDK6 prior the release of Bamboo Cloud on Monday, September 31st. With the end of support for JDK 7 ( deprecated in March 2015 ) also approaching in the upcoming release of Bamboo 5.10 OD5 planned for Sept 28th, we highly recommend upgrading to JDK 8u45 as soon as possible to avoid any issues. Some recommendations: Use JDK 8u45. The latest JDKs are incompatible with some 3rd party libraries we're using. Try to match the layout and scripts of our stock images as closely as possible. This will make it easier for us to provide help if anything goes wrong. Choose Oracle if you have the choice between Oracle and OpenJDK flavor of JDK. We are not able to make Bamboo Cloud compatible with JDK6 due to the nature of the changes recently introduced in the code and custom AMIs need to be upgraded. Please contact our Support Team if you need assistance with the upgrade. Best regards, Sten Pittet Bamboo Product Manager

            There has been no official fix to this in two days, and no announcement of a breaking change, only an upgrade script that ran quietly in the night and assumed all users of the Atlassian product would be using the default configured images (and failed). This managed to push a project of mine outside of the deployment window. I'll be migrating away from Atlassian Cloud Bamboo after this all blows over.

            In any case, thanks for the suggested fix above!

            Steven Thurlow added a comment - There has been no official fix to this in two days, and no announcement of a breaking change, only an upgrade script that ran quietly in the night and assumed all users of the Atlassian product would be using the default configured images (and failed). This managed to push a project of mine outside of the deployment window. I'll be migrating away from Atlassian Cloud Bamboo after this all blows over. In any case, thanks for the suggested fix above!

            Upgrade to java 8 worked for me. adjusted the /etc/profile.d/bamboo.sh to just do that. Also updated my AMI configuration to always follow the lasted AMI Linux image. change /etc/yum.conf and changed the line releasever=2014.03 into releasever=latest. After that: yum clean; yum upgrade. Nice and crispy fresh.

            Toon Leijtens added a comment - Upgrade to java 8 worked for me. adjusted the /etc/profile.d/bamboo.sh to just do that. Also updated my AMI configuration to always follow the lasted AMI Linux image. change /etc/yum.conf and changed the line releasever=2014.03 into releasever=latest. After that: yum clean; yum upgrade. Nice and crispy fresh.

            Neale added a comment -

            We have a custom Windows AMI because we require the Unlimited Strength JCE to be installed.

            Our builds are not running. Artifacts are not publishing and this is not our problem. This is a regression. Can you please update the agent code to revert the breakage. We should have been given plenty of warning of an breaking change like this.

            Neale added a comment - We have a custom Windows AMI because we require the Unlimited Strength JCE to be installed. Our builds are not running. Artifacts are not publishing and this is not our problem. This is a regression. Can you please update the agent code to revert the breakage. We should have been given plenty of warning of an breaking change like this.

            Mike Daly added a comment -

            ...and you likely won't see one until you update your AMI using the workaround listed above. I think it is unlikely that they will fix this issue to work with JDK6. But it sure would have been nice (read: expected) that we would have been informed about this change before it was rolled out.

            Mike Daly added a comment - ...and you likely won't see one until you update your AMI using the workaround listed above. I think it is unlikely that they will fix this issue to work with JDK6. But it sure would have been nice (read: expected) that we would have been informed about this change before it was rolled out.

            Same problem here. I haven't seen a build since the end of August. Please resolve this issue.

            Toon Leijtens added a comment - Same problem here. I haven't seen a build since the end of August. Please resolve this issue.

            Mike Daly added a comment -

            We saw the same behavior. Our AMIs stopped working over the weekend, so I updated to JDK8u60 on our Windows AMIs. Builds were then succeeding but failing to publish any artifacts. I rolled back to JDK8u45 and artifacts started publishing correctly again.

            Mike Daly added a comment - We saw the same behavior. Our AMIs stopped working over the weekend, so I updated to JDK8u60 on our Windows AMIs. Builds were then succeeding but failing to publish any artifacts. I rolled back to JDK8u45 and artifacts started publishing correctly again.

            After upgrading the bamboo agent version to 4.5 and JDK8u45 in windows, the builds are successfully running

            Mafaz Udayar added a comment - After upgrading the bamboo agent version to 4.5 and JDK8u45 in windows, the builds are successfully running

            I am using 8u60 without any issues. For us it's publishing artifacts correctly.

            Jensen Somers added a comment - I am using 8u60 without any issues. For us it's publishing artifacts correctly.

              pbruski Przemek Bruski
              imaduro Ivan Maduro (Inactive)
              Affected customers:
              16 This affects my team
              Watchers:
              31 Start watching this issue

                Created:
                Updated:
                Resolved: