• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • 3.2
    • 3.0.5
    • None
    • None
    • Actual version 3.0.4-studio-4.

      There seems to be some sort of pool exhaustion issue that a Studio customer has hit where the system deadlocks. Thread dumps show hundreds of threads that look like they're waiting for database connections, with nothing obviously running that would free them. The log contains hours of messages from the PreventJobExecutionUntilCompletedTriggerListener about refusing to start a new job due to an increasingly old job (triggered by DEFAULT.elasticAgentMonitorJobTrigger) still going. Situation persisted for several hours until the customer noticed and got support to restart Bamboo.

      Log which includes two thread dumps attached.

            [BAM-9766] Database pool exhaustion leading to deadlock

            The bean that replaced jiraServerManager in 3.2 is non transactional, so the deadlock will not happen.

            Przemek Bruski added a comment - The bean that replaced jiraServerManager in 3.2 is non transactional, so the deadlock will not happen.

            That bean waits for a lock while holding a connection. It only does a simple fetch and does not need a transaction.

            Przemek Bruski added a comment - That bean waits for a lock while holding a connection. It only does a simple fetch and does not need a transaction.

            AntonA added a comment -

            Thanks.

            AntonA added a comment - Thanks.

            Minimum is 0, maximum is 25.

            Hugh Giddens (Inactive) added a comment - Minimum is 0, maximum is 25.

            AntonA added a comment -

            Przemek,

            Could you describe in more detail what the problem is?

            And how your intended fix is going to address it?

            Cheers,
            Anton

            AntonA added a comment - Przemek, Could you describe in more detail what the problem is? And how your intended fix is going to address it? Cheers, Anton

            Przemek Bruski added a comment - - edited

            You can fix it on Studio branch by injecting a non-transactional jiraServerManager into jiraRemoteIssueManager - just split jiraServerManager in to a Tx and non-Tx bean.
            Because of applinks introduction, the code looks very much different in 3.2 and I am not sure whether this can occur. Anyhow, splitting the bean won't hurt and we'll do it.

            Przemek Bruski added a comment - - edited You can fix it on Studio branch by injecting a non-transactional jiraServerManager into jiraRemoteIssueManager - just split jiraServerManager in to a Tx and non-Tx bean. Because of applinks introduction, the code looks very much different in 3.2 and I am not sure whether this can occur. Anyhow, splitting the bean won't hurt and we'll do it.

            jens added a comment -

            We should look into this for 3.3.1 if we don't get around it for 3.0.

            jens added a comment - We should look into this for 3.3.1 if we don't get around it for 3.0.

            AntonA added a comment -

            Thanks for the update.

            Yes, the instance does not seem to be large at all. So hopefully there will be no need to change config.

            What are the current db settings? That is, max and min number of connections allowed.

            We'll try to have a look at it in the next few weeks.

            Cheers,
            Anton

            AntonA added a comment - Thanks for the update. Yes, the instance does not seem to be large at all. So hopefully there will be no need to change config. What are the current db settings? That is, max and min number of connections allowed. We'll try to have a look at it in the next few weeks. Cheers, Anton

            Just a once off; restarting fixed it. They have seven plans, none of which have apparently run in the last two months. We've apparently not increased the number of DB connections for Studio customers in the past; if it turns out to fix the problem a Studio-wide change might be preferred, but hopefully shouldn't be necessary given the small size of the customer's instance.

            Hugh Giddens (Inactive) added a comment - Just a once off; restarting fixed it. They have seven plans, none of which have apparently run in the last two months. We've apparently not increased the number of DB connections for Studio customers in the past; if it turns out to fix the problem a Studio-wide change might be preferred, but hopefully shouldn't be necessary given the small size of the customer's instance.

            AntonA added a comment - - edited

            Hi Hugh,

            We are in the last stretch for Bamboo 3.3, so it may be a few weeks before we look into this.

            Is the customer's Bamboo instance continually running into this, or was this a once off?

            What is the size of the customer's Bamboo instance? That is, how many plans do they have and how many concurrent agents are usually running?

            It may be the case where the maximum allowed number of database connections has to be increased.

            Do you guys to this for Studio customers?

            Cheers,
            Anton

            AntonA added a comment - - edited Hi Hugh, We are in the last stretch for Bamboo 3.3, so it may be a few weeks before we look into this. Is the customer's Bamboo instance continually running into this, or was this a once off? What is the size of the customer's Bamboo instance? That is, how many plans do they have and how many concurrent agents are usually running? It may be the case where the maximum allowed number of database connections has to be increased. Do you guys to this for Studio customers? Cheers, Anton

              pbruski Przemek Bruski
              hgiddens Hugh Giddens (Inactive)
              Affected customers:
              0 This affects my team
              Watchers:
              0 Start watching this issue

                Created:
                Updated:
                Resolved: