Uploaded image for project: 'Jira Platform Cloud'
  1. Jira Platform Cloud
  2. JRACLOUD-67570

JIRA Cloud REST API /rest/api/latest/search?maxResults=1000 is returning only 100 results.

    • Icon: Bug Bug
    • Resolution: Not a bug
    • Icon: High High
    • Ecosystem
    • None

      Status Update

      Hi everyone,

      Let me shed some light behind this issue.

      I confirm that the change of maxResults value for search API was intentional and it’s not a bug. We decided to reduce the maxResults value not only due to increased performance and memory impact on Jira but also after observing the volume of REST API requests that were unfinished because of timeout errors. Whilst we appreciate that the solution may not be ideal to the problem, and we keep investing into improving the performance of Jira searches, it was necessary to reduce the number of failed API requests without further delay.

      According to Atlassian REST API policy default and maximum sizes of paged data are not considered part of the API and may change without deprecation notice. But I completely understand the impact this change caused to REST API clients that relied on the anticipated value of 1000 results and I apologize for lack of prior communication.

      In principle, our recommended solution is to rely on pagination to retrieve the desired number of results in chunks for any API that supports startAt parameter. We also recommend that REST API clients systematically confirm maxResults value when making the requests to prevent disruptions whenever these limits change. That being said, we updated the related Knowledge Base article, which was inaccurately suggesting the default value of response results to be 1000.

      Regarding the problems for startAt offset above 1000 in combination with expanded changelog, we need to investigate them as separate issue (see JRACLOUD-67458).


      Thank you for your understanding.

      Eve Stankiewicz
      Jira Cloud Product Management

      Summary

      JIRA Cloud REST API /rest/api/latest/search?maxResults=1000 is returning only 100 results.

      Steps to Reproduce

      1. Call to a REST API with maxResults = 1000

      Expected Results

      The API returns all issues until 1000 results.

      Actual Results

      The API returns only 100 results.

      Notes:

      This is impacting add-on dashboards and reporting.

      Changing maxResults Parameter for JIRA REST API.

      Workaround

      You can use the startAt parameter to get the other results (if more than 100 are returned). /search endpoint implements pagination api which can be used to retrieve all the required data

            [JRACLOUD-67570] JIRA Cloud REST API /rest/api/latest/search?maxResults=1000 is returning only 100 results.

            Not that pagination and "start-at" mechanisms are inherently broken.... Assume we have items sequentially numbered 1 thru 5000... Pagination is set to 25 items..
            So expectation:
               first call: 1 thru 25
               second call 26 thru 50

            BUT if the collection changes between the calls 
            Say #3 is deleted after the first call.. Then "skipping 25" would result in 27 thru 51 being returned and #26 never beeng seen.

            ATOMIC results require the use of some persistent (Atlassian server side)  snapshot to ensure that all of the (potentially hundereds of sequential reqests) all are related to the data as it exists at a single point in time.

            David V. Corbin added a comment - Not that pagination and "start-at" mechanisms are inherently broken.... Assume we have items sequentially numbered 1 thru 5000... Pagination is set to 25 items.. So expectation:    first call: 1 thru 25    second call 26 thru 50 BUT if the collection changes between the calls  Say #3 is deleted after the first call.. Then "skipping 25" would result in 27 thru 51 being returned and #26 never beeng seen . ATOMIC results require the use of some persistent (Atlassian server side)  snapshot to ensure that all of the (potentially hundereds of sequential reqests) all are related to the data as it exists at a single point in time.

            This is a difficult limitation for our system, which is built on top of Jira systems. The given limitation and reasons are not acceptable. Please consider increasing the limits, or at the very least provide an offset parameter to the API so multiple calls can be a workaround. 

            Hooman Foroughi added a comment - This is a difficult limitation for our system, which is built on top of Jira systems. The given limitation and reasons are not acceptable. Please consider increasing the limits, or at the very least provide an offset parameter to the API so multiple calls can be a workaround. 

            I am really sorry that I migrate from server to cloud. Now it's hard to return. This is not acceptable.

            Florentin Szomoru added a comment - I am really sorry that I migrate from server to cloud. Now it's hard to return. This is not acceptable.

            Jira has tons of basic cloud software issues that other cloud software are offering in the market. THIS IS REALLY VERY BAD APPROACH FROM JIRA

            Deleted Account (Inactive) added a comment - Jira has tons of basic cloud software issues that other cloud software are offering in the market. THIS IS REALLY VERY BAD APPROACH FROM JIRA

            Dario B added a comment -

            david.leal, k4el,

            I understand your point and I agree it would be much more comfortable being able to export everything in a shot. On the other hand I have to agree with the developers that it is very difficult to determine the average amount of data each issue may contain and, since we have seen cases in the past of people literally DoSsing their instances by trying to retrieve way too many issues at the same time, they decided to set some kind of 'safe' limit.

            For more details on this, see:

             

            Finally, there is actually an open Feature Request to have the limit increased that has been created more than one year ago but that got only one vote so far:

            I have linked the feature request to this ticket so that maybe more people will be aware of it and will vote for it, giving it a chance to get back on the developers' table at some point in time.

            Dario B added a comment - david.leal , k4el , I understand your point and I agree it would be much more comfortable being able to export everything in a shot. On the other hand I have to agree with the developers that it is very difficult to determine the average amount of data each issue may contain and, since we have seen cases in the past of people literally DoSsing their instances by trying to retrieve way too many issues at the same time, they decided to set some kind of 'safe' limit. For more details on this, see: https://ecosystem.atlassian.net/browse/DEVHELP-1027 https://ecosystem.atlassian.net/browse/ADDON-230   Finally, there is actually an open Feature Request to have the limit increased that has been created more than one year ago but that got only one vote so far: JRACLOUD-67722 I have linked the feature request to this ticket so that maybe more people will be aware of it and will vote for it, giving it a chance to get back on the developers' table at some point in time.

            Incorrect behavior attributed to an "optional" parameter is just making excuses. Policy that allows you to avoid fixing a product people have paid for is just an excuse with paper work.

            Kael Hammond added a comment - Incorrect behavior attributed to an "optional" parameter is just making excuses. Policy that allows you to avoid fixing a product people have paid for is just an excuse with paper work.

            David Leal added a comment -

            @Dario Bonotto, I guess you understand my point people are very disappointed with this approach. I implemented a REST API solution for exporting 2.6K issues, and because of this 100 record limitation, it takes to me about 5 minutes to get all of them in excel. Using a CSV export option that also has a limit about 1000 (not 10K) it takes much less time, but in order to load the information into excel, I need to build a different logic for downloading the file, uploading, parsing the information and dealing with the fact that the excel columns are not consistent among files, because it create an additional column per Sprint (when the issue has been in more than one sprint) and by labels (when there is more than one). Therefore the CSV file format changes dynamically based on the information downloaded, without changing the columns to export.

            Now I need to deal with the logic for getting more than 1K records using excel export (because of the performance) and the logic for using the REST API for a small amount of data.

            Having a limit of 1K for both ways is at least more consistent, and also having a CSV file format that does not change dynamically based on the number of labels, components or Sprint like it happens now. It makes more difficult the logic for parsing the information. For example, using a String of elements delimited in the same column instead of adding dynamically a new column that is more difficult to handle. 

            David Leal added a comment - @Dario Bonotto, I guess you understand my point people are very disappointed with this approach. I implemented a REST API solution for exporting 2.6K issues, and because of this 100 record limitation, it takes to me about 5 minutes to get all of them in excel. Using a CSV export option that also has a limit about 1000 (not 10K) it takes much less time, but in order to load the information into excel, I need to build a different logic for downloading the file, uploading, parsing the information and dealing with the fact that the excel columns are not consistent among files, because it create an additional column per Sprint (when the issue has been in more than one sprint) and by labels (when there is more than one). Therefore the CSV file format changes dynamically based on the information downloaded, without changing the columns to export. Now I need to deal with the logic for getting more than 1K records using excel export (because of the performance) and the logic for using the REST API for a small amount of data. Having a limit of 1K for both ways is at least more consistent, and also having a CSV file format that does not change dynamically based on the number of labels, components or Sprint like it happens now. It makes more difficult the logic for parsing the information. For example, using a String of elements delimited in the same column instead of adding dynamically a new column that is more difficult to handle. 

            Dario B added a comment - - edited

            david.leal As mentioned in the status update on the top of the description, you can use pagination to get all the results.  A similar approach can be used to get more than 10000 result when exporting to CSV (or the old export to Excel).

            I believe that no doctor will ever give you all your medical records in one single page (unless it is very little data) since there is a limited amount of text you can put on a page before it becomes unreadable. The same concept applies here.

            For details see:

            Dario B added a comment - - edited david.leal  As mentioned in the status update on the top of the description, you can use pagination to get all the results.  A similar approach can be used to get more than 10000 result when exporting to CSV (or the old export to Excel). I believe that no doctor will ever give you all your medical records in one single page (unless it is very little data) since there is a limited amount of text you can put on a page before it becomes unreadable. The same concept applies here. For details see: The Jira Cloud platform REST API - Expansion, pagination, and ordering Export More than 1000 Issues to Excel from Issue Navigator in JIRA Cloud   

            David Leal added a comment -

            How Atlassian can put a restriction like this? it means we don't own our data in Jira. Imaging I am going to the doctor for requesting my medical records and I got the same answer: We can give you only the first two pages of your records but you can visit us every day and we will give you the next two pages until you get all your records. This is insane. We have a limitation for exporting into excel file and now this low limit for the REST-API.

            David Leal added a comment - How Atlassian can put a restriction like this? it means we don't own our data in Jira. Imaging I am going to the doctor for requesting my medical records and I got the same answer: We can give you only the first two pages of your records but you can visit us every day and we will give you the next two pages until you get all your records. This is insane. We have a limitation for exporting into excel file and now this low limit for the REST-API.

            Dario B added a comment - - edited

            Hi mike.loux,

            As mentioned in the documentation the 'isLast' property is optional. This means that is not returned by all the endpoints.

            E.g.
            if I call the search endpoint indeed I don't get it:

            {"expand":"schema,names","startAt":0,"maxResults":50,"total":7,"issues":.....

            While if I call rest/agile/1.0/board/ID/sprint:

            {"maxResults":50,"startAt":0,"isLast":true,"values".......

            However, while checking the documentation I have seen that the example for the 'search' endpoint is wrong. I have opened a request to get the documentation fixed:

             

            Cheers,
            Dario

            Dario B added a comment - - edited Hi mike.loux , As mentioned in the documentation the 'isLast' property is optional. This means that is not returned by all the endpoints. E.g. if I call the search endpoint indeed I don't get it: {"expand":"schema,names","startAt":0,"maxResults":50,"total":7,"issues":..... While if I call rest/agile/1.0/board/ID/sprint: {"maxResults":50,"startAt":0,"isLast":true,"values"....... However, while checking the documentation I have seen that the example for the 'search' endpoint is wrong. I have opened a request to get the documentation fixed: JRACLOUD-70163   Cheers, Dario

              Unassigned Unassigned
              adaluz Angélica Luz
              Affected customers:
              10 This affects my team
              Watchers:
              117 Start watching this issue

                Created:
                Updated:
                Resolved: