Bitbucket AMI: Race condition in startup when attaching a snapshot with a large pre-existing database

XMLWordPrintable

    • Type: Bug
    • Resolution: Fixed
    • Priority: Low
    • 4.6.0
    • Affects Version/s: 3.8.0
    • Component/s: Enterprise
    • None

      Cloudformation init can fail when attaching a snapshot with a large pre-existing database.

      Below are the log files from /var/log/cfn-init.log

      2015-12-29 00:32:03 === BEGIN: service atl-init-10-volume start ===
      2015-12-29 00:32:03 Initialising data volume
      2015-12-29 00:32:03 Disabling atl-init-10-volume for future boots
      2015-12-29 00:32:03 Done disabling atl-init-10-volume for future boots
      2015-12-29 00:32:03 Checking /dev/xvdf to be mounted on /media/atl
      2015-12-29 00:32:03 Using preformatted /dev/xvdf: Linux rev 1.0 ext4 filesystem data, UUID=d66c76e3-97f0-4631-8bf8-ada82859abd9 (extents) (large files) (huge files)
      2015-12-29 00:32:03 Creating /media/atl directory
      2015-12-29 00:32:03 Mounting /dev/xvdf on /media/atl type ext4
      2015-12-29 00:32:03 Creating /media/atl/bitbucket
      2015-12-29 00:32:03 Creating /media/atl/bitbucket/shared
      2015-12-29 00:32:04 === END:   service atl-init-10-volume start ===
      2015-12-29 00:32:04 === BEGIN: service atl-init-20-instance-store start ===
      2015-12-29 00:32:04 Initialising instance store
      2015-12-29 00:32:04 Creating instance store directories for enabled product "Bitbucket"
      2015-12-29 00:32:04 Invoking create-instance-store-dirs on service atl-init-bitbucket
      2015-12-29 00:32:04 === BEGIN: service atl-init-bitbucket create-instance-store-dirs ===
      2015-12-29 00:32:04 Initialising Atlassian Bitbucket
      2015-12-29 00:32:04 Creating /media/ephemeral0/bitbucket
      2015-12-29 00:32:04 Creating /media/ephemeral0/bitbucket/caches
      2015-12-29 00:32:04 Creating /media/ephemeral0/bitbucket/tmp
      2015-12-29 00:32:04 Changing ownership of the contents of /media/ephemeral0/bitbucket to atlbitbucket
      2015-12-29 00:32:04 === END:   service atl-init-bitbucket create-instance-store-dirs ===
      2015-12-29 00:32:04 === END:   service atl-init-20-instance-store start ===
      2015-12-29 00:32:04 === BEGIN: service atl-init-30-db start ===
      2015-12-29 00:32:04 Initialising database
      2015-12-29 00:32:04 Disabling atl-init-30-db for future boots
      2015-12-29 00:32:04 Done disabling atl-init-30-db for future boots
      2015-12-29 00:32:04 Ensuring /var/atlassian/application-data exists
      2015-12-29 00:32:04 Atlassian application data mount /media/atl is present
      2015-12-29 00:32:04 Linking /var/atlassian/application-data/db to /media/atl/db
      2015-12-29 00:32:04 Configuring Postgres to use directory /var/atlassian/application-data/db for data
      2015-12-29 00:32:04 /var/atlassian/application-data/db/ is non-empty, skipping Postgres data initialisation
      2015-12-29 00:32:04 Enabling Postgres for configured run levels
      2015-12-29 00:32:04 Done enabling Postgres for configured run levels
      2015-12-29 00:32:04 Starting Postgres
      Starting postgresql93 service: [  OK  ]
      2015-12-29 00:32:06 === END:   service atl-init-30-db start ===
      2015-12-29 00:32:06 === BEGIN:  service atl-init-40-products start ===
      2015-12-29 00:32:06 Initialising enabled Atlassian products
      2015-12-29 00:32:06 Disabling atl-init-40-products for future boots
      2015-12-29 00:32:06 Done disabling atl-init-40-products for future boots
      2015-12-29 00:32:06 Configuring NGINX at /etc/nginx/nginx.conf
      2015-12-29 00:32:06 Updating NGINX server_name to ec2-54-206-4-43.ap-southeast-2.compute.amazonaws.com 54.206.4.43 10.0.0.119 localhost 127.0.0.1 
      2015-12-29 00:32:06 Reloading NGINX Configuration
      2015-12-29 00:32:06 Initialising enabled product "Bitbucket"
      2015-12-29 00:32:06 Starting service atl-init-bitbucket
      2015-12-29 00:32:06 === BEGIN: service atl-init-bitbucket start ===
      2015-12-29 00:32:06 Initialising Atlassian Bitbucket
      2015-12-29 00:32:06 Creating /var/atlassian/application-data/bitbucket
      2015-12-29 00:32:06 Linking /var/atlassian/application-data/bitbucket/shared to /media/atl/bitbucket/shared
      2015-12-29 00:32:06 Linking /var/atlassian/application-data/bitbucket/caches to /media/ephemeral0/bitbucket/caches
      2015-12-29 00:32:06 Linking /var/atlassian/application-data/bitbucket/tmp to /media/ephemeral0/bitbucket/tmp
      2015-12-29 00:32:06 Creating Bitbucket DB role atlbitbucket
      psql: FATAL:  the database system is starting up
      

            Assignee:
            Cristan Szmajda (Inactive)
            Reporter:
            rikf
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: