Nodes do not join cluster when using the AWS CloudFormation template because of .lock file in shared/home

XMLWordPrintable

    • Type: Bug
    • Resolution: Low Engagement
    • Priority: Medium
    • None
    • Affects Version/s: 5.0.1, 5.3.1, 4.14.9
    • Component/s: None
    • 1
    • Severity 3 - Minor
    • 0

      Summary

      When using the AWS quickstart/CloudFormation template, the nodes do not join the cluster after the Data Center instance has been created.

      Environment

      • AWS

      Steps to Reproduce

      • Use quickstart to launch a new DC instance in AWS

      Expected Results

      • All nodes should join the cluster

      Actual Results

      The below exception is thrown in the atlassian-bitbucket.log file of the node that did not join the cluster:

      2017-09-08 17:59:26,511 INFO  [main]  o.s.boot.SpringApplication 
        .   ____          _            __ _ _
       /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
      ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
       \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
        '  |____| .__|_| |_|_| |_\__, | / / / /
       =========|_|==============|___/=/_/_/_/
       :: Spring Boot ::        (v1.5.3.RELEASE)
      
      
      2017-09-08 17:59:26,787 INFO  [main]  c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on ip-10-0-21-69 with PID 14942 (/opt/atlassian/bitbucket/app/WEB-INF/classes started by atlb
      itbucket in /)
      2017-09-08 17:59:26,788 INFO  [main]  c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
      2017-09-08 17:59:28,198 INFO  [main]  c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.0.1 (a53df94 built on Wed May 10 03:40:47 UTC 2017)
      2017-09-08 17:59:28,198 INFO  [main]  c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_141-b16
      2017-09-08 17:59:30,185 INFO  [main]  c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 4.963 seconds (JVM running for 7.11)
      2017-09-08 17:59:34,663 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
      2017-09-08 17:59:36,174 INFO  [spring-startup]  c.a.s.internal.ApplicationSettings Capturing benchmark home version 5.0.1
      2017-09-08 17:59:40,342 ERROR [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Home directory /media/atl/bitbucket/shared has already been locked by another instance
      2017-09-08 17:59:40,343 WARN  [spring-startup]  o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.Unsat
      isfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.fac
      tory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is com.atlassian.s
      tash.internal.home.HomeLockDeniedException: Home directory '/media/atl/bitbucket/shared' is already locked by another running instance of Bitbucket.
      
      Please stop the other instance of Bitbucket and restart this instance.
      
      If you are absolutely certain that no other instance of Bitbucket is running, please remove the lock file '/media/atl/bitbucket/shared/.lock' and restart this instance.
      2017-09-08 17:59:40,369 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket
      2017-09-08 17:59:40,374 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
      org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exc
      eption is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Invocation of init method fail
      ed; nested exception is com.atlassian.stash.internal.home.HomeLockDeniedException: Home directory '/media/atl/bitbucket/shared' is already locked by another running instance of Bitbucket.
      
      Please stop the other instance of Bitbucket and restart this instance.
      
      If you are absolutely certain that no other instance of Bitbucket is running, please remove the lock file '/media/atl/bitbucket/shared/.lock' and restart this instance.
              at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
              at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
              at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
              at javax.servlet.GenericServlet.init(GenericServlet.java:158)
              at java.lang.Thread.run(Thread.java:748)
              ... 18 frames trimmed
      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Invocation of init method fai
      led; nested exception is com.atlassian.stash.internal.home.HomeLockDeniedException: Home directory '/media/atl/bitbucket/shared' is already locked by another running instance of Bitbucket.
      
      Please stop the other instance of Bitbucket and restart this instance.
      
      If you are absolutely certain that no other instance of Bitbucket is running, please remove the lock file '/media/atl/bitbucket/shared/.lock' and restart this instance.
              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
              ... 5 common frames omitted
      Caused by: com.atlassian.stash.internal.home.HomeLockDeniedException: Home directory '/media/atl/bitbucket/shared' is already locked by another running instance of Bitbucket.
      
      Please stop the other instance of Bitbucket and restart this instance.
      
      If you are absolutely certain that no other instance of Bitbucket is running, please remove the lock file '/media/atl/bitbucket/shared/.lock' and restart this instance.
              at com.atlassian.stash.internal.home.HomeLockAcquirer.onLockDenied(HomeLockAcquirer.java:148)
              at com.atlassian.stash.internal.home.HomeLockAcquirer.lock(HomeLockAcquirer.java:63)
              ... 5 common frames omitted
      

      Notes

      • I was able to replicate this with 4.14.9, 5.0.1 and 5.3.1

      Workaround

      • As the error mentions, if the .lock file in the $BITBUCKET_HOME/shared/ directory is removed and the service on the affected node is restarted, it will join the cluster after startup.

            Assignee:
            Unassigned
            Reporter:
            Nate Hansberry (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: