NOTE: This bug report is for JIRA Software Server. Using JIRA Software Cloud? See the corresponding bug report.

      Atlassian Status as of 11th July 2014

      Dear Customers,

      The JIRA Agile Development and Support teams are currently working on resolving this issue at a high priority. This issue was initially found in JIRA Agile 6.4.0+, the specific cause for this issue found and was resolved in the JIRA Agile 6.4.2 release. It has been however found following to this release that subsequent similar issues have been seen where is it not possible to rank issues in JIRA Agile.

      It is understood that the cause of this issue after the 6.4.2 release is different to the initial bug, and that it may have something to do with specific database configuration or data being in an inconsistent state due to the previous bug. One of these issues is exposed within the Oracle Database server and has been logged in the following bug: GHS-10910

      If customers have run into this problem and are not currently on 6.4.2 it is recommended that they immediately upgrade to the latest version of JIRA Agile as this should resolve their problems.

      If the upgrade does not fix the problem customers will need to trigger a balance, details on how to perform this are specified below.

      If the balance does not work and you are on an Oracle Database please follow the steps below entitled Upgrade Notes for JIRA Agile 6.4.2, it is possible you have run into GHS-10910 please follow that for updates.

      In other cases please contact https://support.atlassian.com, if customers urgently require their JIRA Agile to be functional again please follow the Rollback the plugin steps below.

      Cheers,

      Michael Andreacchio
      JIRA Support Team Lead

      Issues are not able to be ranked in JIRA Agile 6.4.x. Basically, when you try to rank the issues you are hitting into this error message:

      JIRA Agile cannot execute the rank operation. This board has recently been configured to use the Rank field. The system must be re-indexed before you can rank issues. Ask your administrator to perform a manual re-index.

      The following error is thrown in the atlassian-jira.log

      2014-07-02 09:11:26,257 http-bio-8080-exec-97 WARN xxxxxx 551x7454x6 1hz4f54 xxx.xxx.10.xxx,192.xxx.1.12 /secure/QuickCreateIssue.jspa [greenhopper.customfield.lexorank.LexoRankIndexer] Could not retrieve LexoRank value for issue[id=123317]. Indexing max LexoRank value instead.
      2014-07-02 09:11:26,797 http-bio-8080-exec-98 WARN xxxxxx 551x7444x3 1xhugqi xxx.xxx.0.xxx,xxx.xxx.1.xxx /browse/TTPSTN-1054 [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [12070] and issue [122761]
      2014-07-02 09:11:26,801 http-bio-8080-exec-98 WARN xxxxxx 551x7444x3 1xhugqi xxx.156.0.xxx,192.xxx.1.12 /browse/TTPSTN-1054 [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout
      2014-07-02 09:11:26,802 http-bio-8080-exec-98 ERROR xxxxxx 551x7444x3 1xhugqi xxx.156.0.xxx,192.xxx.1.12 /browse/TTPSTN-1054 [jira.web.component.ModuleWebComponentImpl] An exception occured while rendering the web panel: com.pyxis.greenhopper.jira:greenhopper-epics-issue-web-panel (null)
      java.lang.NullPointerException
      	at com.atlassian.greenhopper.web.contextprovider.IssuesInEpicContextProvider.getContextMap(IssuesInEpicContextProvider.java:86)
      	at com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel.getHtml(DefaultWebPanelModuleDescriptor.java:144)
      	at com.atlassian.jira.web.component.ModuleWebComponentImpl.renderModule(ModuleWebComponentImpl.java:87)
      	at com.atlassian.jira.web.component.ModuleWebComponentImpl.renderModuleAndLetNoThrowablesEscape(ModuleWebComponentImpl.java:70)
      	at com.atlassian.jira.web.component.ModuleWebComponentImpl.renderModule(ModuleWebComponentImpl.java:57)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:129)
      	at com.sun.proxy.$Proxy415.renderModule(Unknown Source)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)

      The following error is thrown on the issue view page

      If you're running OnDemand, please contact support.

      If you're using an Oracle database, please check https://jira.atlassian.com/browse/GHS-10910 and see if the issue applies to your database.

      Upgrade Notes for JIRA Agile 6.4.2:

      The upgrade involves upgrading the plugin and then checking the database for certain edge cases that can occur that are broken ranking values. If these edge cases have occurred they need to be fixed in the database, which is described below.

      1. Back up JIRA DB and perform the workaround on a staging server first.
      2. Upgrade JIRA Agile.
      3. Stop JIRA.
      4. Run the below SQL to clear up any improper ranking. This is PostgreSQL (see below for Oracle SQL). It's possible that no rows are deleted and this is not considered to be a problem.
        DELETE FROM "AO_60DB71_LEXORANK" WHERE "RANK" LIKE 'zzzzzz:' AND "ISSUE_ID" != 9223372036854775807 AND "TYPE" = 1;
        
      5. Run the below to identify if there are any problematic rows. If a number higher than 1 is returned they will need to be deleted, however it's possible no rows may be found.
        WITH marker_row_buckets AS (SELECT "TYPE", substring("RANK" from 1 for 1) AS bucket FROM "AO_60DB71_LEXORANK" WHERE "TYPE" IN (0, 2))
        SELECT * FROM "AO_60DB71_LEXORANK" WHERE "TYPE" = 1 AND substring("RANK" from 1 for 1) NOT IN (SELECT bucket FROM marker_row_buckets);
        
      6. Delete any rows from the previous SQL, for example:
        DELETE FROM "AO_60DB71_LEXORANK" WHERE "ID" in (<n id from above>,<n+x from above>);
        
      7. Start JIRA.
      8. Run a GET on <JIRA-BASE-URL>/rest/greenhopper/1.0/lexorank/balance ONCE. This will trigger a background job that will perform a balance (re-distribution of rank values, preserves the rank order of issues);
        Run the REST commands while logged in as a JIRA Administrator. The REST Console can be used to generate the GET command.
      9. Monitor the balance background job by running the below query until all ranks are in only 1 bucket.
        SELECT 'ROWS_IN_BUCKET_0', "FIELD_ID", coalesce(COUNT(1), 0) AS rows_in_bucket FROM "AO_60DB71_LEXORANK" WHERE "RANK" LIKE '0|%' GROUP BY "FIELD_ID"
        UNION
        SELECT 'ROWS_IN_BUCKET_1', "FIELD_ID", coalesce(COUNT(1), 0) AS rows_in_bucket FROM "AO_60DB71_LEXORANK" WHERE "RANK" LIKE '1|%' GROUP BY "FIELD_ID"
        UNION
        SELECT 'ROWS_IN_BUCKET_2', "FIELD_ID", coalesce(COUNT(1), 0) AS rows_in_bucket FROM "AO_60DB71_LEXORANK" WHERE "RANK" LIKE '2|%' GROUP BY "FIELD_ID" ORDER BY "FIELD_ID";
        
      10. Check to see if there are no more scheduled balance jobs by running the below query - it should return no rows. If it does return rows this means the balance is scheduled or running and can be monitored with the above SQL from step 10.
        SELECT * FROM "AO_60DB71_LEXORANKBALANCER";
        
      11. A balance can take quite some time depending on the amount of issues and rank fields. A rough performance function
        amount of issues in system * amount of rank fields * duration of balance operation (100ms) 
      12. Re-index (can be background).
      Oracle SQL

      If using Oracle, ensure the database collation is set as per Connecting JIRA to Oracle. It must be AL32UTF8 - please consult with the Oracle DBA if this needs to be changed.

      SELECT COUNT(*) AS issue_row
      FROM AO_60DB71_LEXORANK
      WHERE TYPE = 1
      AND SUBSTR(RANK, 1, 1) NOT IN (
          SELECT SUBSTR(RANK, 1, 1) AS bucket
          FROM AO_60DB71_LEXORANK
          WHERE TYPE IN (0, 2));
      

      Rollback the plugin

      It is possible to rollback to a previous version of the plugin as per the below steps.

      The only version that can be rolled back to is 6.3.13.1. Any other version will cause significant problems.

      1. Unlock the rank fields as per How to unlock a Locked field.
      2. From JIRA Administration > Issues > Custom Fields (left side menu), rename the rank fields to something else, for example Rank (LexoRank).
      3. Rename the old (Obsolete) fields to their original name 'Rank'.
      4. From JIRA Administration > Manage Add-ons, uninstall JIRA Agile.
      5. Stop JIRA.
      6. Roll-back the JIRA Agile version in the database with the following SQL:
        UPDATE propertynumber SET propertyvalue = 42 WHERE id = (SELECT id FROM propertyentry WHERE property_key = 'GreenHopper.Upgrade.Latest.Upgraded.Version');
        UPDATE propertystring SET propertyvalue = '42' WHERE id = (SELECT id FROM propertyentry WHERE property_key = 'com.pyxis.greenhopper.jira:build');
        
      7. Start JIRA.
      8. Install the previous version of JIRA Agile 6.3.13.1. From JIRA Administration > Add-ons > Manage Add-ons (left side menu), choose 'Upload Add-on' and provide the URL or upload from your computer.
      9. Reindex JIRA.

        1. epic.png
          epic.png
          27 kB
        2. jira_log_whenCreatingNewIssue.txt
          27 kB
        3. JST-105908_jira.lexorank.csv
          180 kB
        4. JST-105922_jira.lexorank.csv
          167 kB

            [JSWSERVER-10872] Unable to rank issues in JIRA Agile

            Keith added a comment -

            This is not resolved.

            Keith added a comment - This is not resolved.

            its 2019 and i still have this problem. Its been 5 years and still not fixed

            Peter Hancock added a comment - its 2019 and i still have this problem. Its been 5 years and still not fixed

            LASER added a comment -

            I have JIRA 7.3.8 and still see same issue. what is the resolution ?

            LASER added a comment - I have JIRA 7.3.8 and still see same issue. what is the resolution ?

            Kim Sullivan added a comment - - edited

            Yesterday I upgraded from Agile 6.6.13 to 6.6.41. Today I noticed extremely slow JIRA response, and lots and lots of thethe following in the log files:

            2014-12-23 13:39:15,638 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332]
            2014-12-23 13:39:15,638 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout
            2014-12-23 13:39:19,679 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332]
            2014-12-23 13:39:19,679 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout
            2014-12-23 13:39:19,679 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankIndexer] Could not retrieve LexoRank value for issue[id=36332]. Indexing max LexoRank value instead.
            2014-12-23 13:39:23,736 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332]
            2014-12-23 13:39:23,736 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout
            2014-12-23 13:39:27,812 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332]
            2014-12-23 13:39:27,812 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout
            2014-12-23 13:39:31,829 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332]
            2014-12-23 13:39:31,829 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout
            

            The SQL queries from above did not return any rows, so I tried running the lexorank indexer but it seems that the Lexorank balancer is stuck (even after a restart) - there are 2 buckets and no progress for more than 60 minutes:

            ROWS_IN_BUCKET_0	10400	9930
            ROWS_IN_BUCKET_1	10400	4423
            
            false	10400	1	1406664091133
            

            Runnign Jira v6.2.5 on PostgreSQL.

            UPDATE: Nevermind, I've been bitten by the collation problem as described in Unable to rank and/or balancer fails to complete due to incorrect database collation, and I found the bug already reported in GHS-11178.

            Kim Sullivan added a comment - - edited Yesterday I upgraded from Agile 6.6.13 to 6.6.41. Today I noticed extremely slow JIRA response, and lots and lots of thethe following in the log files: 2014-12-23 13:39:15,638 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332] 2014-12-23 13:39:15,638 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout 2014-12-23 13:39:19,679 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332] 2014-12-23 13:39:19,679 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout 2014-12-23 13:39:19,679 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankIndexer] Could not retrieve LexoRank value for issue[id=36332]. Indexing max LexoRank value instead. 2014-12-23 13:39:23,736 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332] 2014-12-23 13:39:23,736 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout 2014-12-23 13:39:27,812 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332] 2014-12-23 13:39:27,812 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout 2014-12-23 13:39:31,829 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10400] and issue [36332] 2014-12-23 13:39:31,829 http-bio-80-exec-5 WARN sullivan 819x5624x1 7gnprf 10.214.65.123 /secure/AjaxIssueAction.jspa [greenhopper.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout The SQL queries from above did not return any rows, so I tried running the lexorank indexer but it seems that the Lexorank balancer is stuck (even after a restart) - there are 2 buckets and no progress for more than 60 minutes: ROWS_IN_BUCKET_0 10400 9930 ROWS_IN_BUCKET_1 10400 4423 false 10400 1 1406664091133 Runnign Jira v6.2.5 on PostgreSQL. UPDATE : Nevermind, I've been bitten by the collation problem as described in Unable to rank and/or balancer fails to complete due to incorrect database collation , and I found the bug already reported in GHS-11178 .

            @Mateusz Janiszewski

            SQL_Latin1_General_CP437_CI_AI for MsSQL Server 2012.
            In general, if you follow https://confluence.atlassian.com/display/JIRA/Connecting+JIRA+to+a+Database you can't get it wrong .

            Lukáš Maruniak added a comment - @Mateusz Janiszewski SQL_Latin1_General_CP437_CI_AI for MsSQL Server 2012. In general, if you follow https://confluence.atlassian.com/display/JIRA/Connecting+JIRA+to+a+Database you can't get it wrong .

            mateusz.janiszewski1
            For example, compare required PostgreSQL database collation for JIRA 6.2 and JIRA 6.3:

            Could you please confirm lukas.maruniak1 ?

            Dejan Stojadinović added a comment - mateusz.janiszewski1 For example, compare required PostgreSQL database collation for JIRA 6.2 and JIRA 6.3: https://confluence.atlassian.com/display/JIRA062/Connecting+JIRA+to+PostgreSQL#ConnectingJIRAtoPostgreSQL-1.CreateandconfigurethePostgreSQLdatabase Create a database for JIRA to store issues in (e.g. jiradb) with Unicode collation. Remember this database name, as it will be used to configure JIRA's connection to this database in subsequent steps. CREATE DATABASE jiradb WITH ENCODING 'UNICODE'; Or from the command-line: $ createdb -E UNICODE jiradb https://confluence.atlassian.com/display/JIRA063/Connecting+JIRA+to+PostgreSQL#ConnectingJIRAtoPostgreSQL-1.CreateandconfigurethePostgreSQLdatabase Create a database for JIRA to store issues in (e.g. jiradb) with Unicode collation. Remember this database name, as it will be used to configure JIRA's connection to this database in subsequent steps. CREATE DATABASE jiradb WITH ENCODING 'UNICODE' LC_COLLATE 'C' LC_CTYPE 'C' TEMPLATE template0; Or from the command-line: $ createdb -E UNICODE -l C -T template0 jiradb Could you please confirm lukas.maruniak1 ?

            @Lukáš Maruniak:
            What was the proper database collation setting in your case?
            It seems I have encountered a similar problem...

            Mateusz Janiszewski added a comment - @Lukáš Maruniak: What was the proper database collation setting in your case? It seems I have encountered a similar problem...

            Hi,
            Recently, after update to 6.6.13, we had troubles with issue ranking which was caused by incorrect database collation.
            The only fix was to backup/restore system (with proper database collation setting).

            After upgrade and restore, I can say, that JIRA Agile is working very well. Performance improved rapidly and there are no issues with ranking.
            Good job atlassian .

            Lukáš Maruniak added a comment - Hi, Recently, after update to 6.6.13, we had troubles with issue ranking which was caused by incorrect database collation. The only fix was to backup/restore system (with proper database collation setting). After upgrade and restore, I can say, that JIRA Agile is working very well. Performance improved rapidly and there are no issues with ranking. Good job atlassian .

            MattS added a comment -

            The summary at the top of this issue should really warn people not to upgrade past JIRA Agile 6.3.x until these problems are fixed.

            I upgraded a million issue JIRA to 6.6 with three Global Rank fields instead of the expected one field, due to a broken upgrade that proceeded. The mapping task looked like taking 3 days to run, but the real problem was that any issue data change that wanted to reindex the issue took about 15s and all the light box UIs timed out more often than not.
            Justin at Atlassian Support did a great job helping with this and JIRA is now usable again.

            My suggestions to the GH dev team would be to test JIRA Agile changes on a much larger JIRA instance such as 1M issues, and if you only expect one instance of a certain field type, then check that there is only one instance, and fail to start the add-on if that isn't so.

            MattS added a comment - The summary at the top of this issue should really warn people not to upgrade past JIRA Agile 6.3.x until these problems are fixed. I upgraded a million issue JIRA to 6.6 with three Global Rank fields instead of the expected one field, due to a broken upgrade that proceeded. The mapping task looked like taking 3 days to run, but the real problem was that any issue data change that wanted to reindex the issue took about 15s and all the light box UIs timed out more often than not. Justin at Atlassian Support did a great job helping with this and JIRA is now usable again. My suggestions to the GH dev team would be to test JIRA Agile changes on a much larger JIRA instance such as 1M issues, and if you only expect one instance of a certain field type, then check that there is only one instance, and fail to start the add-on if that isn't so.

            Martin added a comment -

            Martin added a comment - Thank u Ivar. https://confluence.atlassian.com/pages/viewpage.action?pageId=665223710 Solved it.

              Unassigned Unassigned
              adiallo Abdoulaye Kindy Diallo (Inactive)
              Affected customers:
              26 This affects my team
              Watchers:
              91 Start watching this issue

                Created:
                Updated:
                Resolved: