Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-21854

Temporary images created by jfreechart are not cleaned up

      None of our JIRA instances are cleaning up the temporary images created by jfreechart. The tomcat/temp directory of jira.atlassian.com had 350,000 files, for example.

      The files look like:

      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11646 Jul 25 14:17 jfreechart-onetime-949352571854230169.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11952 Jul 25 12:39 jfreechart-onetime-957538706595968267.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11757 Jul 25 04:01 jfreechart-onetime-963208896751366693.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11674 Jul 25 16:13 jfreechart-onetime-969684489244123410.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11646 Jul 25 14:41 jfreechart-onetime-971540390290389047.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com   9370 Jul 24 23:57 jfreechart-onetime-974789924774468593.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com   9321 Jul 25 01:05 jfreechart-onetime-977625683314475437.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com   8245 Jul 25 01:50 jfreechart-onetime-985842978856772519.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com   9321 Jul 25 00:07 jfreechart-onetime-986348626327708513.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11674 Jul 25 18:31 jfreechart-onetime-988462419623200103.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11674 Jul 25 18:19 jfreechart-onetime-993845373684162071.png
      -rw-r--r--  1 j2ee_jira.atlassian.com j2ee_jira.atlassian.com  11779 Jul 25 10:31 jfreechart-onetime-999941953045481311.png
      

      This is a major problem as it's quite possible that a highly utilised server (like JAC or SAC) can die due to lack of disk space.

            [JRASERVER-21854] Temporary images created by jfreechart are not cleaned up

            Any performance impact should be in a good way, we remove a write to disk as well as a read (I/O is a big issue in OD) and remove a browser<-->server round trip.

            Trevor Campbell (Inactive) added a comment - Any performance impact should be in a good way, we remove a write to disk as well as a read (I/O is a big issue in OD) and remove a browser<-->server round trip.

            While there are many reasons the image may not be retrieved and therefore left lying around forever, in my testing it seems there is one major culprit which might be simply addressed.

            When a second application requests a gadget from JIRA, e.g. A JIRA gadget on a Confluence page or even another JIRA's dashboard, the gadget makes two requests for the chart, but naturally enough only displays the one (second) image. The first image is never displayed or deleted. So if you have a chart from another JIRA updating on a wallboard each 5 minutes, then you will add one of these every 5 minutes.

            I have uploaded screenshot, of the network traffic from firebug showing a

            • GET to makeRequest followed by a
            • PUT to makeRequest
              Both generate and return information for a chart, but only the second shart is actually requested.

            You can also see from the traffic, the requests go to my localhost, but the image is being sourced from a different running JIRA application.

            The behaviour is slightly different depending upon the application link configuration and if you are authorised on the end target machine, but the effective result is the same - 2 chart requests.

            Trevor Campbell (Inactive) added a comment - - edited While there are many reasons the image may not be retrieved and therefore left lying around forever, in my testing it seems there is one major culprit which might be simply addressed. When a second application requests a gadget from JIRA, e.g. A JIRA gadget on a Confluence page or even another JIRA's dashboard, the gadget makes two requests for the chart, but naturally enough only displays the one (second) image. The first image is never displayed or deleted. So if you have a chart from another JIRA updating on a wallboard each 5 minutes, then you will add one of these every 5 minutes. I have uploaded screenshot, of the network traffic from firebug showing a GET to makeRequest followed by a PUT to makeRequest Both generate and return information for a chart, but only the second shart is actually requested. You can also see from the traffic, the requests go to my localhost, but the image is being sourced from a different running JIRA application. The behaviour is slightly different depending upon the application link configuration and if you are authorised on the end target machine, but the effective result is the same - 2 chart requests.

            Hi matt.doar, unfortunately no, the regression was fixed by reverting this fix, so this fix will have to be changed so that it doesn't cause the regression at JRA-34911

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - Hi matt.doar , unfortunately no, the regression was fixed by reverting this fix, so this fix will have to be changed so that it doesn't cause the regression at JRA-34911 Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

            MattS added a comment -

            Oswaldo, the regression is marked as Fixed so can the fix for this be used now?

            MattS added a comment - Oswaldo, the regression is marked as Fixed so can the fix for this be used now?

            Hi kjell.lauren@ericsson.com,

            Unfortunately, the initial fix to this issue cause an important regression, which was tracked at JRA-34911, so we had to roll-back the fix.

            At the moment this bug is not in our backlog and has not been scheduled for any upcoming JIRA maintenance as we are currently working through higher priority issues in other areas of JIRA. Having said that, it is still a bug we would like to fix, please watch this issue as I will update it when more information is available.

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - - edited Hi kjell.lauren@ericsson.com , Unfortunately, the initial fix to this issue cause an important regression, which was tracked at JRA-34911 , so we had to roll-back the fix. At the moment this bug is not in our backlog and has not been scheduled for any upcoming JIRA maintenance as we are currently working through higher priority issues in other areas of JIRA. Having said that, it is still a bug we would like to fix, please watch this issue as I will update it when more information is available. Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

            The bug still exists in Jira 6.1.5, any news on when it will be fixed?

            Kjell Lauren added a comment - The bug still exists in Jira 6.1.5, any news on when it will be fixed?

            ifurmanov this will be fixed in JIRA 6.1.1 (see JRA-34911). Downgrading is unfortunately not supported.

            Luis Miranda (Inactive) added a comment - ifurmanov this will be fixed in JIRA 6.1.1 (see JRA-34911 ). Downgrading is unfortunately not supported.

            Thanks!
            Apparently I don't have enough permissions to view issue you mentioned.
            Should I expect fix from Confluence or from JIRA side?
            I just checked jira 6.1 and issue is reproduced there.
            Is it possible to downgrade from jira 6.0.8 to jira 6.0.7?

            Ilya Furmanov added a comment - Thanks! Apparently I don't have enough permissions to view issue you mentioned. Should I expect fix from Confluence or from JIRA side? I just checked jira 6.1 and issue is reproduced there. Is it possible to downgrade from jira 6.0.8 to jira 6.0.7?

            Yes, see linked issue CONFDEV-19885.

            Luis Miranda (Inactive) added a comment - Yes, see linked issue CONFDEV-19885.

            Ilya Furmanov added a comment - - edited

            Luis, could those side effect include inability to display jira gadgets in confluence?

            Ilya Furmanov added a comment - - edited Luis, could those side effect include inability to display jira gadgets in confluence?

            Unfortunately the previous changes have caused some unintended side-effects, so I am having to roll back the change.

            Luis Miranda (Inactive) added a comment - Unfortunately the previous changes have caused some unintended side-effects, so I am having to roll back the change.

            Luis Miranda (Inactive) added a comment - - edited

            You will need to clean up the Tomcat temporary directory when you upgrade to 6.0.8 (in fact, you may clean it up periodically but not when JIRA is running).

            why not just stream the files directly, instead of saving them on disk at all?

            That's a good idea, but it would be a bigger scoped change to how charts work in JIRA, and would require a lot more testing/time than this surgical fix.

            Luis Miranda (Inactive) added a comment - - edited You will need to clean up the Tomcat temporary directory when you upgrade to 6.0.8 (in fact, you may clean it up periodically but not when JIRA is running). why not just stream the files directly, instead of saving them on disk at all? That's a good idea, but it would be a bigger scoped change to how charts work in JIRA, and would require a lot more testing/time than this surgical fix.

            Sounds good! What about existing installs where we have 10's of thousands of these files? Will they be deleted, or do we have to delete those manually?

            Also, why not just stream the files directly, instead of saving them on disk at all? Something like:

            <img src="https://example.com/path/to/image_generator?somekey=abc.......efg"/>

            joshua kugler added a comment - Sounds good! What about existing installs where we have 10's of thousands of these files? Will they be deleted, or do we have to delete those manually? Also, why not just stream the files directly, instead of saving them on disk at all? Something like: <img src="https://example.com/path/to/image_generator?somekey=abc.......efg"/>

            I've made a fix for this in JIRA 6.0.8: the jfreechart-onetime-* files created for a user are now also deleted when the user's session is destroyed (either due to logging out or session expiry, which by default is after 5 hours).

            The happy path is still that the charts are deleted after they are streamed once, but failing that they will not outlive the user's session.

            Luis Miranda (Inactive) added a comment - I've made a fix for this in JIRA 6.0.8: the jfreechart-onetime-* files created for a user are now also deleted when the user's session is destroyed (either due to logging out or session expiry, which by default is after 5 hours). The happy path is still that the charts are deleted after they are streamed once, but failing that they will not outlive the user's session.

            This is also happening in JIRA 5.1.5. Any chances that this would be fixed? Have someone tried Luis Miranda's suggestion above?

            Deleted Account (Inactive) added a comment - This is also happening in JIRA 5.1.5. Any chances that this would be fixed? Have someone tried Luis Miranda's suggestion above?

            I'm not convinced JIRA should be reaching into Tomcat's temporary directory and deleting things. I would prefer to configure Tomcat to use $TMPDIR in Unix and %TEMP% in Windows as the location for its temporary files.

            This can be done by adding a few lines to bin/setenv.sh:

            # use the OS location for temporary files if available
            if [ -n "$TMPDIR" -a -z "$CATALINA_TMPDIR" ] ; then
                CATALINA_TMPDIR="$TMPDIR"
                export CATALINA_TMPDIR
            fi
            

            Luis Miranda (Inactive) added a comment - I'm not convinced JIRA should be reaching into Tomcat's temporary directory and deleting things. I would prefer to configure Tomcat to use $TMPDIR in Unix and %TEMP% in Windows as the location for its temporary files. This can be done by adding a few lines to bin/setenv.sh : # use the OS location for temporary files if available if [ -n "$TMPDIR" -a -z "$CATALINA_TMPDIR" ] ; then CATALINA_TMPDIR= "$TMPDIR" export CATALINA_TMPDIR fi

            MattS added a comment -

            I've got 0.5GB of jfreechart debris in my temp directory after a year or so. Seems like a slow killer to me. Why can't JIRA clean the Tomcat temp directory when it starts up?

            MattS added a comment - I've got 0.5GB of jfreechart debris in my temp directory after a year or so. Seems like a slow killer to me. Why can't JIRA clean the Tomcat temp directory when it starts up?

            David Yu added a comment -

            9.5GBs of files in our temp directory (JIRA 5.0.3).

            Mostly jfreechart and some mail attachments as well.

            David Yu added a comment - 9.5GBs of files in our temp directory (JIRA 5.0.3). Mostly jfreechart and some mail attachments as well.

            If the chart is never served, then it is up to the usual OS-level mechanisms that purge /tmp.

            As Arnaud says, your assumption above is incorrect. The temp files are created in the app server's temp directory by default, which has no OS-level mechanism to purge, and if it did it varies with OS. JIRA should be cleaning up it's own temporary files.

            Pramod Korathota (Inactive) added a comment - If the chart is never served, then it is up to the usual OS-level mechanisms that purge /tmp. As Arnaud says, your assumption above is incorrect. The temp files are created in the app server's temp directory by default, which has no OS-level mechanism to purge, and if it did it varies with OS. JIRA should be cleaning up it's own temporary files.

            Arnaud Heritier added a comment - - edited

            I'm not really agree with your comment.
            I just had the same issue today (jira 5.0.5) and it is a pain to remove these files because the OS doesn't want to do a rm when the number is too large (I had to play with find + xargs).
            tmp directory isn't necessarily the OS one but in my case the app server one which has no automated deletion system.
            This constraint must a least be documented for Jira administrators to avoid surprizes.

            Arnaud Heritier added a comment - - edited I'm not really agree with your comment. I just had the same issue today (jira 5.0.5) and it is a pain to remove these files because the OS doesn't want to do a rm when the number is too large (I had to play with find + xargs). tmp directory isn't necessarily the OS one but in my case the app server one which has no automated deletion system. This constraint must a least be documented for Jira administrators to avoid surprizes.

            JFreeChart deletes onetime charts after they are served up:

                    if (isChartInUserList || isChartPublic || isOneTimeChart) {
                        //  Serve it up
                        ServletUtilities.sendTempFile(file, response);
                        if (isOneTimeChart) {
                            file.delete();
                        }
                    }
            

            If the chart is never served, then it is up to the usual OS-level mechanisms that purge /tmp.

            Luis Miranda (Inactive) added a comment - JFreeChart deletes onetime charts after they are served up: if (isChartInUserList || isChartPublic || isOneTimeChart) { // Serve it up ServletUtilities.sendTempFile(file, response); if (isOneTimeChart) { file.delete(); } } If the chart is never served, then it is up to the usual OS-level mechanisms that purge /tmp.

            Looking at the temp dir on JAC it only has files from the last 5 hours, so it looks to be fixed:

            [pkorathota@atlassian temp]$ ls -latr jfreechart-onetime-*| awk {'print $6, $7, $8'} | awk -F: '{print $1":"}' | uniq -c
                 94 May 15 15:
                110 May 15 16:
                113 May 15 17:
                101 May 15 18:
                 49 May 15 19:
            

            5 hours is better than never, but on a busy server that can still generate a LOT of files.

            Pramod Korathota (Inactive) added a comment - Looking at the temp dir on JAC it only has files from the last 5 hours, so it looks to be fixed: [pkorathota@atlassian temp]$ ls -latr jfreechart-onetime-*| awk {'print $6, $7, $8'} | awk -F: '{print $1":"}' | uniq -c 94 May 15 15: 110 May 15 16: 113 May 15 17: 101 May 15 18: 49 May 15 19: 5 hours is better than never, but on a busy server that can still generate a LOT of files.

            JFreechart was upgraded in 4.4, worth retesting

            Peter Leschev added a comment - JFreechart was upgraded in 4.4, worth retesting

              ohernandez@atlassian.com Oswaldo Hernandez (Inactive)
              pkorathota Pramod Korathota (Inactive)
              Affected customers:
              18 This affects my team
              Watchers:
              29 Start watching this issue

                Created:
                Updated:
                Resolved: