Uploaded image for project: 'Confluence Cloud'
  1. Confluence Cloud
  2. CONFCLOUD-53651

In-App Notification Is Repeatedly Causing Out Of Memory Exception

      Recently, the following bug was fixed:
      https://jira.atlassian.com/browse/CONF-34439

      A similar and possibly related bug is happening where In-app notification for a page is causing an instance to have an out of memory exception, be restarted and have the out of memory exception again.

      The following is a thread dump for the problem:

      Suspicious Thread: 0x8a90a920
         at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
        at java.lang.StringCoding$StringDecoder.decode([BII)[C (StringCoding.java:149)
        at java.lang.StringCoding.decode(Ljava/lang/String;[BII)[C (StringCoding.java:193)
        at java.lang.String.<init>([BIILjava/lang/String;)V (String.java:414)
        at org.postgresql.core.Encoding.decode([BII)Ljava/lang/String; (Encoding.java:193)
        at org.postgresql.core.Encoding.decode([B)Ljava/lang/String; (Encoding.java:205)
        at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(I)Ljava/lang/String; (AbstractJdbc2ResultSet.java:1879)
        at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(Ljava/lang/String;)Ljava/lang/String; (AbstractJdbc2ResultSet.java:2405)
        at org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.getString(Ljava/lang/String;)Ljava/lang/String; (DelegatingResultSet.java:263)
        at org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.getString(Ljava/lang/String;)Ljava/lang/String; (DelegatingResultSet.java:263)
        at
      ...
      com.atlassian.mywork.host.dao.ao.AONotificationDao.queryNotifications(Lnet/java/ao/Query;)Ljava/lang/Iterable; (AONotificationDao.java:274)
        at com.atlassian.mywork.host.dao.ao.AONotificationDao.findAllUnreadAfterOnlyIdsAction(Ljava/lang/String;J)Ljava/lang/Iterable; (AONotificationDao.java:223)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Method.java:483)
      

      The page responsible for this problem seems to be a regular page.

      Disabling In-app notification stopped the exception from happening all the time.

            [CONFCLOUD-53651] In-App Notification Is Repeatedly Causing Out Of Memory Exception

            Minh Tran added a comment -

            Dear frederic.esnault,

            I think the problem you mentioned related to this issue https://jira.atlassian.com/browse/CONF-38615. It has been fixed in the latest cloud and will be released in 5.10.0

            Thanks,
            Minh Tran
            Confluence BugMaster
            Atlassiano

            Minh Tran added a comment - Dear frederic.esnault , I think the problem you mentioned related to this issue https://jira.atlassian.com/browse/CONF-38615 . It has been fixed in the latest cloud and will be released in 5.10.0 Thanks, Minh Tran Confluence BugMaster Atlassiano

            Hi Minh Tran and thanks for your answer.

            I do not have any browser version / os / screenshots to provide, but I may have a lead on this.

            I made some tests in psql to check what the AO_9412A1_AONOTIFICATION table was containing.
            First, when i just issue a simple "select * from AO_9412A1_AONOTIFICATION", PostgreSQL takes a long time to start displaying any answer. The table only contains 6832 lines.

            After that, I tried displaying only one line :

            confluence=>  select * from "AO_9412A1_AONOTIFICATION" OFFSET 1 LIMIT 1;
            confluence=>  select * from "AO_9412A1_AONOTIFICATION" OFFSET 15 LIMIT 1;
            confluence=>  select * from "AO_9412A1_AONOTIFICATION" OFFSET 2000 LIMIT 1;
            ...
            

            For some lines (very few), I had a normal output, like this (some data deleted/modified as they are customer data) :

             ACTION  |       ACTION_ICON_URL        |             APPLICATION             |         APPLICATION_LINK_ID          |         CREATED         |      DESCRIPTION      |
             ENTITY |                            GLOBAL_ID                             | GROUPING_ID |              ICON_URL              |  ID   |                           ITEM_I
            CON_URL                            |                     ITEM_TITLE                      |     ITEM_URL     |                                                           
                METADATA                                                               | PINNED | READ | STATUS |                TITLE                 |         UPDATED         |  
                                    URL                          |               USER               
            ---------+------------------------------+-------------------------------------+--------------------------------------+-------------------------+-----------------------+
            --------+------------------------------------------------------------------+-------------+------------------------------------+-------+---------------------------------
            -----------------------------------+-----------------------------------------------------+------------------+-----------------------------------------------------------
            ---------------------------------------------------------------------------+--------+------+--------+--------------------------------------+-------------------------+--
            -----------------------------------------------------+----------------------------------
             comment | /images/icons/comment_16.gif | com.atlassian.mywork.providers.jira | c8fd6b9a-5657-3530-a8e7-ebb7c8709f71 | 2016-03-24 09:55:49.038 | <p>Under analysis</p> |
             issue  | appId=c8fd6b9a-5657-3530-a8e7-ebb7c8709f71&entity=issue&id=14180 |             | /secure/useravatar?ownerId=tstark | 48837 | /secure/viewavatar?size=xsmall&a
            vatarId=10320&avatarType=issuetype | TTT-2799: [CUSTOMER] 202 error has returned | /browse/TTT-2799 | {"key":"TTT-2799","summary":"[CUSTOMER] 202 error  
            has returned","user":"Tony STARK","username":"tstark","watching":true} | f      | t    | DONE   | Tonhy Stark commented on an issue. | 2016-03-24 09:55:49.038 | /
            browse/TTT-2799?focusedCommentId=49154#comment-49154 | 2c9596814dd3c2ea014dfc402dee0013
            (1 row)
            

            But on some other lines (quite a few actually), the output is huge, covering several pages (psql pagination, hit space several times, and did not go to the end, I stopped before).
            At first it looked like there could be a problem with PostgreSQL, as some data was looking like system logs, or html code.
            But when I checked one of the issues mentionned in one of the long outputs, it turns out that the users have :

            • Added some huge logs (copy/paste in a comment)
            • Attached some huge log files.

            The thing is that everything is copied in PostgreSQL notifications table (including the full comment or the full attached files), in PostgreSQL fields of type 'text', which means they are not limited in size.
            And of course, when trying to load multiple lines like this, the application quickly runs out of memory, as it creates String objects for all the huge logs files, huge comments, or huge whatever.

            In my opinion, there is a problem with users behavior, as they should not attach so huge files/comments in a jira issue, but there should be a limitation to the notification sent to confluence.
            For example, instead of a notification showing the full comment or attached file, you could only save a very limited size of the notification target (first line of comment or first line of file - if readable), and propose that the users clicks the provided issue link to see the full issue.

            So to reproduce, i think you could just create a few issues in a jira instance linked to a confluence instance, then add huge files (1 or 2Mb text files), to them, copy their contents in comments, then from time to time, reload a confluence page so that notifications are reloaded). Of course you must be able to see the issues as a jira users (maybe be a watcher?), otherwise you won't receive notifications in confluence. I don't know what quantity of data or number of issues is necessary to create the problem, but eventually, Confluence will run out of memory. You could just give it a very little Xms/Xmx to speed up the problem.

            Let me know if you need more information.

            Regards,
            Frederic Esnault

            Frédéric Esnault added a comment - Hi Minh Tran and thanks for your answer. I do not have any browser version / os / screenshots to provide, but I may have a lead on this. I made some tests in psql to check what the AO_9412A1_AONOTIFICATION table was containing. First, when i just issue a simple "select * from AO_9412A1_AONOTIFICATION", PostgreSQL takes a long time to start displaying any answer. The table only contains 6832 lines. After that, I tried displaying only one line : confluence=> select * from "AO_9412A1_AONOTIFICATION" OFFSET 1 LIMIT 1; confluence=> select * from "AO_9412A1_AONOTIFICATION" OFFSET 15 LIMIT 1; confluence=> select * from "AO_9412A1_AONOTIFICATION" OFFSET 2000 LIMIT 1; ... For some lines (very few), I had a normal output, like this (some data deleted/modified as they are customer data) : ACTION | ACTION_ICON_URL | APPLICATION | APPLICATION_LINK_ID | CREATED | DESCRIPTION | ENTITY | GLOBAL_ID | GROUPING_ID | ICON_URL | ID | ITEM_I CON_URL | ITEM_TITLE | ITEM_URL | METADATA | PINNED | READ | STATUS | TITLE | UPDATED | URL | USER ---------+------------------------------+-------------------------------------+--------------------------------------+-------------------------+-----------------------+ --------+------------------------------------------------------------------+-------------+------------------------------------+-------+--------------------------------- -----------------------------------+-----------------------------------------------------+------------------+----------------------------------------------------------- ---------------------------------------------------------------------------+--------+------+--------+--------------------------------------+-------------------------+-- -----------------------------------------------------+---------------------------------- comment | /images/icons/comment_16.gif | com.atlassian.mywork.providers.jira | c8fd6b9a-5657-3530-a8e7-ebb7c8709f71 | 2016-03-24 09:55:49.038 | <p>Under analysis</p> | issue | appId=c8fd6b9a-5657-3530-a8e7-ebb7c8709f71&entity=issue&id=14180 | | /secure/useravatar?ownerId=tstark | 48837 | /secure/viewavatar?size=xsmall&a vatarId=10320&avatarType=issuetype | TTT-2799: [CUSTOMER] 202 error has returned | /browse/TTT-2799 | { "key" : "TTT-2799" , "summary" :"[CUSTOMER] 202 error has returned "," user ":" Tony STARK "," username ":" tstark "," watching": true } | f | t | DONE | Tonhy Stark commented on an issue. | 2016-03-24 09:55:49.038 | / browse/TTT-2799?focusedCommentId=49154#comment-49154 | 2c9596814dd3c2ea014dfc402dee0013 (1 row) But on some other lines (quite a few actually), the output is huge, covering several pages (psql pagination, hit space several times, and did not go to the end, I stopped before). At first it looked like there could be a problem with PostgreSQL, as some data was looking like system logs, or html code. But when I checked one of the issues mentionned in one of the long outputs, it turns out that the users have : Added some huge logs (copy/paste in a comment) Attached some huge log files. The thing is that everything is copied in PostgreSQL notifications table (including the full comment or the full attached files), in PostgreSQL fields of type 'text', which means they are not limited in size. And of course, when trying to load multiple lines like this, the application quickly runs out of memory, as it creates String objects for all the huge logs files, huge comments, or huge whatever. In my opinion, there is a problem with users behavior, as they should not attach so huge files/comments in a jira issue, but there should be a limitation to the notification sent to confluence. For example, instead of a notification showing the full comment or attached file, you could only save a very limited size of the notification target (first line of comment or first line of file - if readable), and propose that the users clicks the provided issue link to see the full issue. So to reproduce, i think you could just create a few issues in a jira instance linked to a confluence instance, then add huge files (1 or 2Mb text files), to them, copy their contents in comments, then from time to time, reload a confluence page so that notifications are reloaded). Of course you must be able to see the issues as a jira users (maybe be a watcher?), otherwise you won't receive notifications in confluence. I don't know what quantity of data or number of issues is necessary to create the problem, but eventually, Confluence will run out of memory. You could just give it a very little Xms/Xmx to speed up the problem. Let me know if you need more information. Regards, Frederic Esnault

            Minh Tran added a comment -

            Dear frederic.esnault],

            Thanks for submitting the ticket
            Would you please let me know steps to reproduce (including the browser name + operating system) and screenshots of the error you met?

            Thanks,
            Minh Tran
            Confluence BugMaster
            Atlassian

            Minh Tran added a comment - Dear frederic.esnault ], Thanks for submitting the ticket Would you please let me know steps to reproduce (including the browser name + operating system) and screenshots of the error you met? Thanks, Minh Tran Confluence BugMaster Atlassian

            Hi Xavier,
            We encountered this issue in a production Confluence. Two times in two days, our Confluence ran out of memory due to the very same problem. The only plugin installed is Team Calendars.

             -- url: /rest/mywork/1/notification 
            2016-03-24 07:26:22,901 ERROR [http-bio-8090-exec-242] [mywork.host.rest.ExceptionLogger] toResponse Exception thrown from resource
             -- referer: https://my.services.ingenico.com/calendar/mycalendar.action | url: /rest/mywork/latest/status/notification/count 
            com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
            Database:
                    - name:PostgreSQL
                    - version:9.3.4
                    - minor version:3
                    - major version:9
            Driver:
                    - name:PostgreSQL Native Driver
                    - version:PostgreSQL 9.2 JDBC4 (build 1002)
            
            org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
                at com.atlassian.mywork.host.dao.ao.AONotificationDao.queryNotifications(AONotificationDao.java:274)
                at com.atlassian.mywork.host.dao.ao.AONotificationDao.findAllUnreadAfterOnlyIdsAction(AONotificationDao.java:223)
            

            Could you please reopen this ticket and try to fix this ?

            Currently I disabled In-App notifications, as I saw that it would help avoid the problem, but this must be a temporary workaround (if working).

            Regards,
            Frederic Esnault

            Frédéric Esnault added a comment - Hi Xavier, We encountered this issue in a production Confluence. Two times in two days, our Confluence ran out of memory due to the very same problem. The only plugin installed is Team Calendars. -- url: / rest /mywork/1/notification 2016-03-24 07:26:22,901 ERROR [http-bio-8090-exec-242] [mywork.host. rest .ExceptionLogger] toResponse Exception thrown from resource -- referer: https: //my.services.ingenico.com/calendar/mycalendar.action | url: / rest /mywork/latest/status/notification/count com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library: Database: - name:PostgreSQL - version:9.3.4 - minor version:3 - major version:9 Driver: - name:PostgreSQL Native Driver - version:PostgreSQL 9.2 JDBC4 (build 1002) org.postgresql.util.PSQLException: Ran out of memory retrieving query results. at com.atlassian.mywork.host.dao.ao.AONotificationDao.queryNotifications(AONotificationDao.java:274) at com.atlassian.mywork.host.dao.ao.AONotificationDao.findAllUnreadAfterOnlyIdsAction(AONotificationDao.java:223) Could you please reopen this ticket and try to fix this ? Currently I disabled In-App notifications, as I saw that it would help avoid the problem, but this must be a temporary workaround (if working). Regards, Frederic Esnault

              xtaixe Xavier Sanchez
              jsilveira Jaime S
              Affected customers:
              0 This affects my team
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: