Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-42282

On large instances the createmeta REST endpoint can take a very long amount of time to return results or cause JIRA to OOM

XMLWordPrintable

    • 6.03
    • 51
    • Severity 1 - Critical
    • 110
    • Hide
      Atlassian Update - 2 September 2019

      Dear users,

      Createmeta REST endpoint is commonly used when Atlassian products (Confluence, Bitbucket, and others) create issues in Jira. It might also be used by 3rd party apps.

      The main problem with the old createmeta endpoint is that it is duplicating values in the response. Although the response can be filtered, due to the design of the endpoint it is very easy to cause downtime.

      We have been working on a fix for this problem most of the solutions made the endpoint harder and more complex to use. So we decided to deprecate the old endpoint and introduce new endpoints that will replace the old createmeta endpoint. We also added a new dark feature to disable the old endpoint. The new endpoints will return projects, issue types and fields separately. You can find more details here.

      Changes will be released with Jira 8.4.0 and the old createmeta will be removed in Jira 9. We suggest anyone who is using the old endpoint to migrate to the new endpoints.

      Since this endpoint is used by other Atlassian products, they will be updated also. We are planning the changes and we will update this ticket with the compatible version of each product.

      Sincerely,

      Celebi Murat
      Jira Server Development

      Show
      Atlassian Update - 2 September 2019 Dear users, Createmeta REST endpoint is commonly used when Atlassian products (Confluence, Bitbucket, and others) create issues in Jira. It might also be used by 3rd party apps. The main problem with the old createmeta endpoint is that it is duplicating values in the response. Although the response can be filtered, due to the design of the endpoint it is very easy to cause downtime. We have been working on a fix for this problem most of the solutions made the endpoint harder and more complex to use. So we decided to deprecate the old endpoint and introduce new endpoints that will replace the old createmeta endpoint. We also added a new dark feature to disable the old endpoint. The new endpoints will return projects, issue types and fields separately. You can find more details here . Changes will be released with Jira 8.4.0 and the old createmeta will be removed in Jira 9. We suggest anyone who is using the old endpoint to migrate to the new endpoints. Since this endpoint is used by other Atlassian products, they will be updated also. We are planning the changes and we will update this ticket with the compatible version of each product. Sincerely, Celebi Murat Jira Server Development

      NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

      Summary

      Executing requests to /jira/rest/api/2/issue/createmeta?params on larger instances can take over a minute to complete and generate large content (500MB+). In some cases JIRA can run out of memory or have heavy GC pressure. At this stage we believe there are 2 possible problems:

      1. In case of standard requests with projects.issuetypes parameters, problem is caused by duplicated data for issue-types, see Duplicate content below for details.
      2. With fields added (createmeta?projects.issuetypes.fields), it's more likely that JIRA can run out of memory. The main problem is that objects representing either issue types or custom fields are not reused among projects. So theoretically this issue could be reproduced with one multi-select custom field with thousands of values and a thousand of projects.

      Environment

      The following data exists in the test instance:

      Database Statistics
      Issues          417547
      Projects        1481
      Custom Fields   1203
      Workflows       324
      Attachments     0
      Comments        2921131
      Users           17225
      Groups          985
      

      And 323 issue types.

      For projects.issuetypes.fields paraments

      Having instance with multi-select CF with large number of values magnifies the problem.
      So problem is a multiplication of factors:

      • Number of CustomFields values x Number of IssueTypes x Number of Projects having that

      Steps to Reproduce

      1. Setup or use an instance with similar data sizes.
      2. Query /rest/api/2/issue/createmeta?projects.issuetypes, for example https://jira.atlassian.com/rest/api/2/issue/createmeta?projects.issuetypes.

      Expected Results

      The query returns in a reasonable timeframe. No JVM memory pressure.

      Actual Results

      The query takes ~60 secs as per the attached screenshot, or in some cases runs out of memory entirely.

      Possible cause and room for improvement

      We decided to remove this endpoint in Jira 9.0 see Createmeta REST endpoint to be removed

      Duplicate content

      We have seen that in some cases there are logical duplicates in the optionconfiguration table. This, depending on the number of affected issue types, can cause performance issues itself. To check for such, you can run the following query:

      // Check duplicated issuetype, query should return 0
      select optionid,fieldconfig,count(1) from optionconfiguration where fieldid = 'issuetype' group by optionid,fieldconfig having count (1) > 1;
      
      // Check all duplicated options, query should return 0
      SELECT fieldid,optionid,fieldconfig, count(*) FROM optionconfiguration group by fieldid,optionid,fieldconfig having count(*)>1
      

      If the above returns any rows, you are likely affected by the problem. In this case, please raise a ticket on https://support.atlassian.com, referring to this bug report and ask for assistance on de-duplicating the table.
      See related JRASERVER-66306

      Workarounds

      • Block the call to REST API at proxy level
        • Any integration using this API will be affected, (eg: create JIRA issue from Confluence will not work)
      • Increasing the heap could be strategy for small heaps (< 10GB)
      Specific workarounds for projects.issuetypes.fields

      REST createmeta will show all data to create issue, so you remove the CF from screen it will be not present in the payload for that specific issues.

      • If possible, delete unused CustomFields and/or CustomFields values.
        • Make CustomFields more specific:
          • Choose applicable issue types
          • Choose applicable context (Project)
        • Remove the CustomFields from screen which are not require it.

        1. Slow createmeta.jpg
          Slow createmeta.jpg
          1.17 MB
        2. Create_Meta.png
          Create_Meta.png
          198 kB

              cmurat Celebi Murat (Inactive)
              ayakovlev@atlassian.com Andriy Yakovlev [Atlassian]
              Votes:
              14 Vote for this issue
              Watchers:
              43 Start watching this issue

                Created:
                Updated:
                Resolved: