Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-3643

Many many open activeMQ threads consume 800% cpu on beac

XMLWordPrintable

      BEAC has been pegged for over a week. It looks like there are many activeMQ threads running in a tight loop

        1. beac-thread-dump.txt
          419 kB
        2. beac-thread-dump-20090313.txt
          467 kB
        3. beac-threaddump-20090408.txt
          510 kB

            [BAM-3643] Many many open activeMQ threads consume 800% cpu on beac

            MarkC added a comment -

            By "now" I mean "next release" which is 2.3-m2

            MarkC added a comment - By "now" I mean "next release" which is 2.3-m2

            David Cheney (Inactive) added a comment -

            Nope, still occuring

            David Cheney (Inactive) added a comment - Nope, still occuring

            MarkC added a comment - - edited

            Looks like the CPU spike was due to the serverside tunnel connection reads was happening in a tight loop if an agent is brought down but the instance stays. The connection leak may be related to to the tunnel not being shutdown properly when the Instance went down. Will need to confirm this once we get the next milestone up on BEAC.

            MarkC added a comment - - edited Looks like the CPU spike was due to the serverside tunnel connection reads was happening in a tight loop if an agent is brought down but the instance stays. The connection leak may be related to to the tunnel not being shutdown properly when the Instance went down. Will need to confirm this once we get the next milestone up on BEAC.

            MarkC added a comment -

            Nah 2.2 should have the fix in it. So something is still not working right

            MarkC added a comment - Nah 2.2 should have the fix in it. So something is still not working right

            David Cheney (Inactive) added a comment -

            Interesting. beac is running 2.2 build 1025, which milestone does that
            relate to, it looks like Adrian fixed this problem in m5.

            On 13/03/2009, at 2:19 PM, Mark Chaimungkalanont [Atlassian] (JIRA)

            David Cheney (Inactive) added a comment - Interesting. beac is running 2.2 build 1025, which milestone does that relate to, it looks like Adrian fixed this problem in m5. On 13/03/2009, at 2:19 PM, Mark Chaimungkalanont [Atlassian] (JIRA)

            MarkC added a comment -

            May be a reappearance of BAM-3300

            MarkC added a comment - May be a reappearance of BAM-3300

            MarkC added a comment -

            Each agent may have up to 8 connections open to the broker (validly) so that may be the issue.

            The original description of "Purging build results & CPU" is definitely a problem. I'm not too sure what the current problem is, but we'll try to take a peek.

            MarkC added a comment - Each agent may have up to 8 connections open to the broker (validly) so that may be the issue. The original description of "Purging build results & CPU" is definitely a problem. I'm not too sure what the current problem is, but we'll try to take a peek.

            George Barnett added a comment -

            spoem:~ gbarnett$ grep 'daemon prio=' tmp | grep Active | sed -e 's/.tcp/tcp/g' -e 's/:....." daemon.//g' | sort | uniq -c | sort -rn
            36 tcp:///172.16.3.27
            24 tcp:///172.16.3.25
            24 tcp:///127.0.0.1
            12 tcp:///192.168.3.145
            12 tcp:///172.16.3.30
            12 tcp:///172.16.3.29
            12 tcp:///172.16.3.230
            12 tcp:///172.16.3.221
            12 tcp:///172.16.3.216
            12 tcp:///172.16.3.210
            12 tcp:///172.16.3.206
            12 tcp:///172.16.3.202
            4 tcp:///172.16.3.209

            This would appear to show bamboo leaking an active mq thread for every agent that comes and goes. Bad for EC2.

            George Barnett added a comment - spoem:~ gbarnett$ grep 'daemon prio=' tmp | grep Active | sed -e 's/. tcp/tcp/g' -e 's/:....." daemon. //g' | sort | uniq -c | sort -rn 36 tcp:///172.16.3.27 24 tcp:///172.16.3.25 24 tcp:///127.0.0.1 12 tcp:///192.168.3.145 12 tcp:///172.16.3.30 12 tcp:///172.16.3.29 12 tcp:///172.16.3.230 12 tcp:///172.16.3.221 12 tcp:///172.16.3.216 12 tcp:///172.16.3.210 12 tcp:///172.16.3.206 12 tcp:///172.16.3.202 4 tcp:///172.16.3.209 This would appear to show bamboo leaking an active mq thread for every agent that comes and goes. Bad for EC2.

            David Cheney (Inactive) added a comment -

            Added recent thread dump

            David Cheney (Inactive) added a comment - Added recent thread dump

              mark@atlassian.com MarkC
              dcheney David Cheney (Inactive)
              Affected customers:
              0 This affects my team
              Watchers:
              0 Start watching this issue

                Created:
                Updated:
                Resolved: