JIRA
  1. JIRA
  2. JRA-25836

Mails are not sent after upgrade from 4.4 to 4.4.1 on Redhat in a Virtual Machine

    Details

      Description

      After installation of version 4.4.1 (was 4.4), the notification emails rest in the mail queue and are not sent anymore. When clicking on "Flush mail queue", all mails are sent immediately without any problems. No errormessage (including OutOfMemoryException) to find in the logfiles.

        Issue Links

          Activity

          Jan Lichter created issue -
          Hide
          Jan Sören Ramm added a comment -

          I've got the same Problem.

          Show
          Jan Sören Ramm added a comment - I've got the same Problem.
          James Winters [Atlassian] made changes -
          Field Original Value New Value
          Status Open [ 1 ] Verified [ 10005 ]
          James Winters [Atlassian] made changes -
          Status Verified [ 10005 ] Awaiting Development [ 10038 ]
          SteakHolder jwinters
          Fix Version/s Bugfix Release [ 15240 ]
          James Winters [Atlassian] made changes -
          Labels todo
          James Winters [Atlassian] made changes -
          Link This issue is duplicated by JRA-25787 [ JRA-25787 ]
          Hide
          pigot nicolas added a comment -

          Same issue for me. If I restart, the flush works for about 1 day and then stop.

          Show
          pigot nicolas added a comment - Same issue for me. If I restart, the flush works for about 1 day and then stop.
          Robert Smart [Atlassian] made changes -
          Global Rank Ranked higher
          Hide
          Brooke Hedrick added a comment -

          This is what we observed in working on the issue.

          1. Mail server configurations lost port settings and protocols after and/or during the import of the data.
          2. JEMH was not working and required the 0.9.8-1 upgrade due to changes in the CSVReader class.
          3. RAM was not increased. Our settings already have a max heap setting of 1024m.

          The mail systems are working at this time but we have not had them operational for 24 hours.
          Others have experienced mail system that work for up to 24 hours and then fail.
          We are monitoring the situation.

          Show
          Brooke Hedrick added a comment - This is what we observed in working on the issue. 1. Mail server configurations lost port settings and protocols after and/or during the import of the data. 2. JEMH was not working and required the 0.9.8-1 upgrade due to changes in the CSVReader class. 3. RAM was not increased. Our settings already have a max heap setting of 1024m. The mail systems are working at this time but we have not had them operational for 24 hours. Others have experienced mail system that work for up to 24 hours and then fail. We are monitoring the situation.
          Robert Smart [Atlassian] made changes -
          Global Rank Ranked lower
          Robert Smart [Atlassian] made changes -
          Global Rank Ranked lower
          Hide
          Robert Smart [Atlassian] added a comment -

          Hi all,

          We have confirmed this is occurring with RedHat linux when run on a virtual machine, there is a known kernel bug in this scenario that was causing the scheduled job for sending mail from the mail queue to never be fired.

          Please see: http://confluence.atlassian.com/x/Iw76Dw for a more detailed explanation.

          Here is the info on the redhat bug:

          Host OS : RedHat 5
          Guest OS : RedHat 5 32b
          Virtualisation software : VMWare ESX

          Content of the issue :

          RHEL5 Running as a VMware Guest

          By default, the timer interrupt (tick) rate in Red Hat Enterprise Linux 4 and 5 is 1000 Hz. In some situations,
          this rate is too high, with frequent interrupts potentially causing a performance impact or unexpected behavior.
          This impact would be most notable when running Red Hat Enterprise Linux as a virtual guest. For example,
          with the 1000 Hz tick rate, time drift might be observed in a virtual environment.
          To compensate for the fact that different workloads and different environments work better with different tick
          rates, a new kernel parameter was added to Red Hat Enterprise Linux 5.1 and 4.7. This kernel parameter, "divider",
          is a boot-time setting that allows the user to divide the tick rate by the desired amount. For example, setting
          the divider to 10 would cause the default tick rate to be 1000/10, or 100 Hz.
          The divider parameter can be set by appending "divider=N" to the kernel boot line in /boot/grub/grub.-
          conf.
          The recommended kernel line settings for 64-bit Red Hat Enterprise Linux 5 running as a VMware guest are:
          divider=10 notsc iommu=soft elevator=noop
          and for a 32-bit Red Hat Enterprise Linux 5 Vmware guest:
          divider=10 clocksource=acpi_pm iommu=soft elevator=noop

          Regards,

          Robert Smart

          Show
          Robert Smart [Atlassian] added a comment - Hi all, We have confirmed this is occurring with RedHat linux when run on a virtual machine, there is a known kernel bug in this scenario that was causing the scheduled job for sending mail from the mail queue to never be fired. Please see: http://confluence.atlassian.com/x/Iw76Dw for a more detailed explanation. Here is the info on the redhat bug: Host OS : RedHat 5 Guest OS : RedHat 5 32b Virtualisation software : VMWare ESX Content of the issue : RHEL5 Running as a VMware Guest By default, the timer interrupt (tick) rate in Red Hat Enterprise Linux 4 and 5 is 1000 Hz. In some situations, this rate is too high, with frequent interrupts potentially causing a performance impact or unexpected behavior. This impact would be most notable when running Red Hat Enterprise Linux as a virtual guest. For example, with the 1000 Hz tick rate, time drift might be observed in a virtual environment. To compensate for the fact that different workloads and different environments work better with different tick rates, a new kernel parameter was added to Red Hat Enterprise Linux 5.1 and 4.7. This kernel parameter, "divider", is a boot-time setting that allows the user to divide the tick rate by the desired amount. For example, setting the divider to 10 would cause the default tick rate to be 1000/10, or 100 Hz. The divider parameter can be set by appending "divider=N" to the kernel boot line in /boot/grub/grub.- conf. The recommended kernel line settings for 64-bit Red Hat Enterprise Linux 5 running as a VMware guest are: divider=10 notsc iommu=soft elevator=noop and for a 32-bit Red Hat Enterprise Linux 5 Vmware guest: divider=10 clocksource=acpi_pm iommu=soft elevator=noop Regards, Robert Smart
          Robert Smart [Atlassian] made changes -
          Status Awaiting Development [ 10038 ] Open [ 1 ]
          Robert Smart [Atlassian] made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Resolved Locally [ 7 ]
          Hide
          Bogdan Dziedzic [Atlassian] added a comment - - edited

          Users who experience 'time drifting' behaviour on their virtualised platform are encouraged to seek assistance from a provided of their vm solution or online resources.

          Related bug report for RedHat 307201 and as per Red Hat Enterprise Linux 5.4 kernel security and bug fix update the situation should be improved when newer kernel is used (kernel-2.6.18-164.el5+).

          A quick search on the Internet brings a few good resources re this matter:

          Show
          Bogdan Dziedzic [Atlassian] added a comment - - edited Users who experience 'time drifting' behaviour on their virtualised platform are encouraged to seek assistance from a provided of their vm solution or online resources. Related bug report for RedHat 307201 and as per Red Hat Enterprise Linux 5.4 kernel security and bug fix update the situation should be improved when newer kernel is used (kernel-2.6.18-164.el5+). A quick search on the Internet brings a few good resources re this matter: Timekeeping best practices for Linux guests Timekeeping in VMware Virtual Machines
          Hide
          Brooke Hedrick added a comment -

          I am confused. This was not an issue in prior versions of Jira running on the exact same O/S. In my case, fixing the class loading issue with JEMH resolve all mail issues. It's as if some part of the mail system can stop working and then all mail services fail. It appears that the mail service may track that it is trying to send email, when there a failure, the mail service then is stuck "sending mail" permanently.

          We made no O/S changes and our problem resolved.

          Show
          Brooke Hedrick added a comment - I am confused. This was not an issue in prior versions of Jira running on the exact same O/S. In my case, fixing the class loading issue with JEMH resolve all mail issues. It's as if some part of the mail system can stop working and then all mail services fail. It appears that the mail service may track that it is trying to send email, when there a failure, the mail service then is stuck "sending mail" permanently. We made no O/S changes and our problem resolved.
          Hide
          Robert Smart [Atlassian] added a comment -

          Hi Brooke,

          This unfortunately is not the problem you were seeing. I am glad to see you could solve your problem locally. It looks to me like you were hit with a different problem to the one that customers were experiencing here.

          Cheers,

          Robert Smart.
          bugwrangler.

          Show
          Robert Smart [Atlassian] added a comment - Hi Brooke, This unfortunately is not the problem you were seeing. I am glad to see you could solve your problem locally. It looks to me like you were hit with a different problem to the one that customers were experiencing here. Cheers, Robert Smart. bugwrangler.
          Hide
          Jan Lichter added a comment -

          Hi,

          now i am confused, too. We are running Jira (4.4.1) on a Suse Enterprise Server 10, physical machine, no vm. And we have also the class loading problem:

          Exception in thread "QuartzWorker-1" java.lang.NoClassDefFoundError: com/mindprod/csv/CSVReader
          at com.javahollic.jira.emh.processor.CSVFieldProcessor.getFieldMap(CSVFieldProcessor.java:64)
          at com.javahollic.jira.emh.processor.CSVFieldProcessor.extractFieldsAndBody(CSVFieldProcessor.java:176)
          at com.javahollic.jira.emh.processor.CSVFieldProcessor.getDirectiveCount(CSVFieldProcessor.java:218)
          at com.javahollic.jira.emh.service.EMHIssueHandler.handleMessage(EMHIssueHandler.java:116)
          at com.javahollic.jira.emh.service.CreateOrCommentHandler.handleMessage(CreateOrCommentHandler.java:87)
          at com.atlassian.jira.service.services.mail.MailFetcherService.run(MailFetcherService.java:186)
          at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60)
          at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)
          at org.quartz.core.JobRunShell.run(JobRunShell.java:195)
          at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72)
          at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
          Caused by: java.lang.ClassNotFoundException: com.mindprod.csv.CSVReader
          at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
          at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
          ... 11 more

          though we have the latest version of the jemh-plugin (0.9.8-1)!?

          Any Idea? Original issue with logs was JSP-92922.

          Best Regards, Jan

          Show
          Jan Lichter added a comment - Hi, now i am confused, too. We are running Jira (4.4.1) on a Suse Enterprise Server 10, physical machine, no vm. And we have also the class loading problem: Exception in thread "QuartzWorker-1" java.lang.NoClassDefFoundError: com/mindprod/csv/CSVReader at com.javahollic.jira.emh.processor.CSVFieldProcessor.getFieldMap(CSVFieldProcessor.java:64) at com.javahollic.jira.emh.processor.CSVFieldProcessor.extractFieldsAndBody(CSVFieldProcessor.java:176) at com.javahollic.jira.emh.processor.CSVFieldProcessor.getDirectiveCount(CSVFieldProcessor.java:218) at com.javahollic.jira.emh.service.EMHIssueHandler.handleMessage(EMHIssueHandler.java:116) at com.javahollic.jira.emh.service.CreateOrCommentHandler.handleMessage(CreateOrCommentHandler.java:87) at com.atlassian.jira.service.services.mail.MailFetcherService.run(MailFetcherService.java:186) at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60) at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47) at org.quartz.core.JobRunShell.run(JobRunShell.java:195) at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520) Caused by: java.lang.ClassNotFoundException: com.mindprod.csv.CSVReader at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526) ... 11 more though we have the latest version of the jemh-plugin (0.9.8-1)!? Any Idea? Original issue with logs was JSP-92922. Best Regards, Jan
          Hide
          Wojciech Seliga added a comment -

          @Jan Lichter: I think you are facing this NoClassDefFoundError issue due to JRA-25315 - that's the removal of mindprod library from JIRA installation.
          Removing this in a bugfix release was apparently not a good idea from our side - apologies.
          As a quick solution I think that taking mindprod.jar from an earlier JIRA version (up to 4.4) from <inst dir>/atlassian-jira/WEB-INF/lib and copying it to the same directory in the new installation (JIRA 4.4.1+) should most probably resolve this very problem you have.

          Cheers,
          Wojtek

          Show
          Wojciech Seliga added a comment - @Jan Lichter: I think you are facing this NoClassDefFoundError issue due to JRA-25315 - that's the removal of mindprod library from JIRA installation. Removing this in a bugfix release was apparently not a good idea from our side - apologies. As a quick solution I think that taking mindprod.jar from an earlier JIRA version (up to 4.4) from <inst dir>/atlassian-jira/WEB-INF/lib and copying it to the same directory in the new installation (JIRA 4.4.1+) should most probably resolve this very problem you have. Cheers, Wojtek
          Hide
          Brooke Hedrick added a comment -

          Jan,

          I can tell you that JEMH 0.9.8-1 definitely contains the fix for the classloading issue.

          I would NOT dig up the old class and put it in place as suggested by support, if I had your issue. This will just cause support to tell you that what you did is not supported and possible make it harder to work other support cases.

          If you open the jemh-mail-handler-0.9.8-1.jar, find the class com.javahollic.jira.emh.processor.CSVFieldProcessor and open it with something like jdgui ( Java Decompiler), you will see the following import:
          import au.com.bytecode.opencsv.CSVReader;

          Noticed it is not trying to import: com/mindprod/csv/CSVReader

          The version of JEMH just before this one, does not have the updated import.

          I suspect either you are on the previous version of the jar or maybe when you tried to fix the issue, you left the previous JEMH jar file in the classpath. Since the jars have the version number in them, it would be easy to accidently leave the old version in place when you copy the new version in.

          BTH - just an Atlassian customer!

          Show
          Brooke Hedrick added a comment - Jan, I can tell you that JEMH 0.9.8-1 definitely contains the fix for the classloading issue. I would NOT dig up the old class and put it in place as suggested by support, if I had your issue. This will just cause support to tell you that what you did is not supported and possible make it harder to work other support cases. If you open the jemh-mail-handler-0.9.8-1.jar, find the class com.javahollic.jira.emh.processor.CSVFieldProcessor and open it with something like jdgui ( Java Decompiler), you will see the following import: import au.com.bytecode.opencsv.CSVReader; Noticed it is not trying to import: com/mindprod/csv/CSVReader The version of JEMH just before this one, does not have the updated import. I suspect either you are on the previous version of the jar or maybe when you tried to fix the issue, you left the previous JEMH jar file in the classpath. Since the jars have the version number in them, it would be easy to accidently leave the old version in place when you copy the new version in. BTH - just an Atlassian customer!
          Hide
          Brooke Hedrick added a comment - - edited

          Robert,

          RE: my issue not being the one on the ticket. This is where support directed me. If I am on the wrong ticket, maybe support was wrong to send me here.

          Also, the description of this issue leaves it open to the possibility that there are multiple causes.

          I definitely was seeing: "After installation of version 4.4.1 (was 4.4), the notification emails rest in the mail queue and are not sent anymore. When clicking on "Flush mail queue", all mails are sent immediately without any problems. No errormessage (including OutOfMemoryException) to find in the logfiles."

          Show
          Brooke Hedrick added a comment - - edited Robert, RE: my issue not being the one on the ticket. This is where support directed me. If I am on the wrong ticket, maybe support was wrong to send me here. Also, the description of this issue leaves it open to the possibility that there are multiple causes. I definitely was seeing: "After installation of version 4.4.1 (was 4.4), the notification emails rest in the mail queue and are not sent anymore. When clicking on "Flush mail queue", all mails are sent immediately without any problems. No errormessage (including OutOfMemoryException) to find in the logfiles."
          Chris Mountford [Atlassian] made changes -
          Summary Mails are not sent after upgrade from 4.4 to 4.4.1 Mails are not sent after upgrade from 4.4 to 4.4.1 on Redhat in a Virtual Machine
          Chris Mountford [Atlassian] made changes -
          Labels todo linux virtualization
          Robert Smart [Atlassian] made changes -
          Link This issue is duplicated by JRA-26519 [ JRA-26519 ]
          Partha Kamal [Atlassian] made changes -
          Remote Link This issue links to "Wiki Page (Extranet)" [ 12078 ]
          Oswaldo Hernandez [Atlassian] Bugmaster made changes -
          Workflow JIRA Bug Workflow w Kanban v5 [ 346369 ] JIRA Bug Workflow w Kanban v6 [ 662973 ]
          Oswaldo Hernandez [Atlassian] Bugmaster made changes -
          Workflow JIRA Bug Workflow w Kanban v5 [ 662973 ] JIRA Bug Workflow w Kanban v6 [ 670614 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Jan Lichter
            • Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: