Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-19964

Bamboo does not repeat unpause task after failure in backup job

XMLWordPrintable

      Summary

      During Bamboo internal backup function, if the one of the clustered database shutdown for maintenance, Bamboo does not repeat the unpause task when it manage to reconnect to one of the clustered database

      Steps to Reproduce

      1. Connect Bamboo to a clustered of two Oracle databases
      2. Set a Backup Scheduler in Bamboo
      3. During backup job running, shutdown one of the clustered Oracle database to cause error to the backup job

      Expected Results

      The backup job fail and Bamboo will unpause the server

      Actual Results

      The backup job fail and Bamboo try to unpause the server but fail with the following error:

      2018-06-28 01:16:55,804 WARN [scheduler_Worker-4] [SqlExceptionHelper] SQL Error: 1089, SQLState: 08006
2018-06-28 01:16:55,804 ERROR [scheduler_Worker-4] [SqlExceptionHelper] ORA-01089: immediate shutdown or close in progress - no operations are permitted

2018-06-28 01:16:55,812 INFO [scheduler_Worker-4] [XmlMigrator] Ensuring the the file has been unmounted
...
2018-06-28 01:16:55,815 WARN [scheduler_Worker-4] [NewPooledConnection] [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
2018-06-28 01:16:55,815 WARN [scheduler_Worker-4] [NewPooledConnection] [c3p0] Another error has occurred [ java.sql.SQLRecoverableException: ORA-01089: immediate shutdown or close in progress - no operations are permitted
 ] which will not be reported to listeners!
java.sql.SQLRecoverableException: ORA-01089: immediate shutdown or close in progress - no operations are permitted
...
2018-06-28 01:16:55,844 ERROR [scheduler_Worker-4] [ScheduledBackupJob] java.lang.IllegalStateException: Could not transition server from 'PAUSING' to 'RUNNING'
java.lang.IllegalStateException: Could not transition server from 'PAUSING' to 'RUNNING'
	at com.atlassian.bamboo.ServerLifecycleManagerImpl.updateServerLifecycleState(ServerLifecycleManagerImpl.java:284)
	at com.atlassian.bamboo.ServerLifecycleManagerImpl.serverRunning(ServerLifecycleManagerImpl.java:132)
	at com.atlassian.bamboo.ServerLifecycleManagerImpl.resumeServer(ServerLifecycleManagerImpl.java:244)
	at com.atlassian.bamboo.migration.ImportExportManagerImpl.bambooExport(ImportExportManagerImpl.java:96)
	at com.atlassian.bamboo.configuration.ScheduledBackupJob.execute(ScheduledBackupJob.java:71)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: org.springframework.transaction.TransactionSystemException: Could not commit Hibernate transaction; nested exception is org.hibernate.TransactionException: Unable to commit against JDBC Connection

      However, it does not repeat the process to unpause Bamboo after it manage to connect to the second clustered Oracle database

      Notes

      Bamboo should repeat the task to unpause server when it manage to connect to the database again

            [BAM-19964] Bamboo does not repeat unpause task after failure in backup job

            Martyna Wojtas (Inactive) made changes -
            Resolution New: Timed out [ 10 ]
            Status Original: Gathering Impact [ 12072 ] New: Closed [ 6 ]
            Martyna Wojtas (Inactive) made changes -
            Labels New: bamboo-bugs-grooming
            Pawel Skierczynski made changes -
            Status Original: Needs Triage [ 10030 ] New: Gathering Impact [ 12072 ]
            Monique Khairuliana (Inactive) made changes -
            Workflow Original: Bamboo Workflow 2016 v1 - Restricted [ 2706400 ] New: JAC Bug Workflow v3 [ 3386060 ]
            Status Original: Open [ 1 ] New: Needs Triage [ 10030 ]
            Bugfix Automation Bot made changes -
            Support reference count Original: 1 New: 2
            Owen made changes -
            Symptom Severity Original: Major [ 14431 ] New: Severity 2 - Major [ 15831 ]
            SET Analytics Bot made changes -
            Support reference count New: 1
            Krystian Brazulewicz made changes -
            Occurrence Factor New: 10% [ 12830 ]
            Symptom Severity Original: Minor [ 14432 ] New: Major [ 14431 ]
            Status Original: Needs Triage [ 10030 ] New: Open [ 1 ]
            Krystian Brazulewicz made changes -
            Component/s New: Database (Oracle) [ 12661 ]
            Foong (Inactive) made changes -
            Affects Version/s New: 6.3.3 [ 78522 ]
            Description New: h3. Summary
            During Bamboo internal backup function, if the one of the clustered database shutdown for maintenance, Bamboo does not repeat the unpause task when it manage to reconnect to one of the clustered database

            h3. Steps to Reproduce
            # Connect Bamboo to a clustered of two Oracle databases
            # Set a Backup Scheduler in Bamboo
            # During backup job running, shutdown one of the clustered Oracle database to cause error to the backup job

            h3. Expected Results
            The backup job fail and Bamboo will unpause the server

            h3. Actual Results
            The backup job fail and Bamboo try to unpause the server but fail with the following error:
            {noformat}
            2018-06-28 01:16:55,804 WARN [scheduler_Worker-4] [SqlExceptionHelper] SQL Error: 1089, SQLState: 08006
            2018-06-28 01:16:55,804 ERROR [scheduler_Worker-4] [SqlExceptionHelper] ORA-01089: immediate shutdown or close in progress - no operations are permitted

            2018-06-28 01:16:55,812 INFO [scheduler_Worker-4] [XmlMigrator] Ensuring the the file has been unmounted
            ...
            2018-06-28 01:16:55,815 WARN [scheduler_Worker-4] [NewPooledConnection] [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
            2018-06-28 01:16:55,815 WARN [scheduler_Worker-4] [NewPooledConnection] [c3p0] Another error has occurred [ java.sql.SQLRecoverableException: ORA-01089: immediate shutdown or close in progress - no operations are permitted
             ] which will not be reported to listeners!
            java.sql.SQLRecoverableException: ORA-01089: immediate shutdown or close in progress - no operations are permitted
            ...
            2018-06-28 01:16:55,844 ERROR [scheduler_Worker-4] [ScheduledBackupJob] java.lang.IllegalStateException: Could not transition server from 'PAUSING' to 'RUNNING'
            java.lang.IllegalStateException: Could not transition server from 'PAUSING' to 'RUNNING'
            at com.atlassian.bamboo.ServerLifecycleManagerImpl.updateServerLifecycleState(ServerLifecycleManagerImpl.java:284)
            at com.atlassian.bamboo.ServerLifecycleManagerImpl.serverRunning(ServerLifecycleManagerImpl.java:132)
            at com.atlassian.bamboo.ServerLifecycleManagerImpl.resumeServer(ServerLifecycleManagerImpl.java:244)
            at com.atlassian.bamboo.migration.ImportExportManagerImpl.bambooExport(ImportExportManagerImpl.java:96)
            at com.atlassian.bamboo.configuration.ScheduledBackupJob.execute(ScheduledBackupJob.java:71)
            at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
            at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
            at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
            at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
            at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
            at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
            Caused by: org.springframework.transaction.TransactionSystemException: Could not commit Hibernate transaction; nested exception is org.hibernate.TransactionException: Unable to commit against JDBC Connection
            {noformat}
            However, it does not repeat the process to unpause Bamboo after it manage to connect to the second clustered Oracle database
             
            h3. Notes
            Bamboo should repeat the task to unpause server when it manage to connect to the database again
            Foong (Inactive) created issue -

              Unassigned Unassigned
              klfoong Foong (Inactive)
              Affected customers:
              1 This affects my team
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: