Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-11178

Unable to rank with any database collation that does not order all alphanumeric characters the same as ASCII

      Description

      This bug is caused by specific collation type such as Danish/Norwegian, Swedish/Finish treat the alphabetical ordering quite differently. For example, in Danish/Norwegian two letters "aa" are treated as a single letter, and comes after "z" in the alphabet. This causes the lexorank ordering to be wrong.

      Symptoms

      The following error message is seen in JIRA Agile when attempting an issue ranking operation.

      Diagnosis

        1. Check your collation type if it affects alphabetical ordering, ie Danish/Norwegian
        2. Run the following query :
          SELECT 'ROWS_IN_BUCKET_0', "FIELD_ID", coalesce(COUNT(1), 0) AS rows_in_bucket FROM "AO_60DB71_LEXORANK" WHERE "RANK" LIKE '0|%' GROUP BY "FIELD_ID"
          UNION
          SELECT 'ROWS_IN_BUCKET_1', "FIELD_ID", coalesce(COUNT(1), 0) AS rows_in_bucket FROM "AO_60DB71_LEXORANK" WHERE "RANK" LIKE '1|%' GROUP BY "FIELD_ID"
          UNION
          SELECT 'ROWS_IN_BUCKET_2', "FIELD_ID", coalesce(COUNT(1), 0) AS rows_in_bucket FROM "AO_60DB71_LEXORANK" WHERE "RANK" LIKE '2|%' GROUP BY "FIELD_ID" ORDER BY "FIELD_ID";
          
        3. The results should show that there is a balancing going on (there will be 2 rows of results, with a number of issues in one bucket and another number in the other bucket)
        4. Enable DEBUG Logging for the balancer : com.atlassian.greenhopper.service.lexorank.balance
        5. From there, we can see that there is something wrong with the balancer. Notably, this line:
          New rank[1|hzzzvz:] for issue[id=31321] for rank field[id=10890] already exists, retrying balance oepration
          

          This error shows that the Balancer is trying to insert a rank value that already exists.

      Workarounds

      Please refer to Unable to rank and/or balancer fails to complete due to incorrect database collation

          Form Name

            [JSWSERVER-11178] Unable to rank with any database collation that does not order all alphanumeric characters the same as ASCII

            Kim Sullivan added a comment - - edited

            I recently upgraded to Agile 6.6.41 and I encountered this exact problem. It seems that your collation checks don't detect all special cases, in Czech collation, CH comes after H. I had to set the log to DEBUG, because there was no warning message like described above.

            For PostgreSQL (I have 9.3), the workaround seems to be this (the workaround in the KB article only mentions changing the collation for the whole database which was not acceptable for me):

            ALTER TABLE "AO_60DB71_LEXORANK"
               ALTER COLUMN "RANK" TYPE character varying(255) COLLATE pg_catalog."C";
            

            Kim Sullivan added a comment - - edited I recently upgraded to Agile 6.6.41 and I encountered this exact problem. It seems that your collation checks don't detect all special cases, in Czech collation, CH comes after H. I had to set the log to DEBUG, because there was no warning message like described above. For PostgreSQL (I have 9.3), the workaround seems to be this (the workaround in the KB article only mentions changing the collation for the whole database which was not acceptable for me): ALTER TABLE "AO_60DB71_LEXORANK" ALTER COLUMN " RANK " TYPE character varying (255) COLLATE pg_catalog. " C " ;

            OK. Wasn't even aware that you could change collation on a single table. This seems like a really dangerous solution/workaround to me.. Perhaps changing the collation of the entire database would/could be a better workaround but will effectively kill the ability to use local language in the application (which is kind of a bum).

            Filip Strandqvist added a comment - OK. Wasn't even aware that you could change collation on a single table. This seems like a really dangerous solution/workaround to me.. Perhaps changing the collation of the entire database would/could be a better workaround but will effectively kill the ability to use local language in the application (which is kind of a bum).

            The problem disappeared for us after applying the workaround (we have Swedish Finnish collation), but I'm not sure if it's a good practice to have a different collation on one specific table.

            I think it would be better if Atlassian would fix the bug by rewriting Agile so that it can properly handle different collations,

            Oskar Hanberg added a comment - The problem disappeared for us after applying the workaround (we have Swedish Finnish collation), but I'm not sure if it's a good practice to have a different collation on one specific table. I think it would be better if Atlassian would fix the bug by rewriting Agile so that it can properly handle different collations,

            I believe we are seeing the effect of this to. We run JIRA 6.3.11 and Agile 6.6.13 (with Swedish/Finnish collation). We experience (as an example) QuickCreateIssue execution times of some 20 seconds. Many other parts of the system performs slowly as well. We have a minima, out-of-the-box installation (quite fresh) on good, powerful hardware, with very few add-ons. This could not possibly be "normal" with JIRA, could it?

            Filip Strandqvist added a comment - I believe we are seeing the effect of this to. We run JIRA 6.3.11 and Agile 6.6.13 (with Swedish/Finnish collation). We experience (as an example) QuickCreateIssue execution times of some 20 seconds. Many other parts of the system performs slowly as well. We have a minima, out-of-the-box installation (quite fresh) on good, powerful hardware, with very few add-ons. This could not possibly be "normal" with JIRA, could it?

            Is it really considered "resolved" if performance is affected? How much is performance affected and when?

            Oskar Hanberg added a comment - Is it really considered "resolved" if performance is affected? How much is performance affected and when?

            This issue should now be resolved in JIRA Agile 6.6.13. What we have done to fix this is to put in place additional checks to figure out what collation is currently being used by the underlying database. If those checks tell us that the collation is non-standard, we apply additional sorting functions to our SQL queries when performing rank operations, to ensure that our ranking algorithm still performs as expected.

            In this situation however, performance will be impacted, as applying these additional sorting functions slows down the operation of the database. There is also a message output in the logs when we detect this case:

            ****************************************************************************************************
             The database collation is set incorrectly for JIRA Agile. This can cause JIRA Agile to run slower. 
             If you are using PostgreSQL 9.0 or earlier, this may also cause ranking to fail.
             Please refer to the following documentation on the database configuration: https://confluence.atlassian.com/display/DOC/Database+Configuration
             If the problems persist, please refer to this KB on how to fix the database: https://confluence.atlassian.com/x/PAQWK
            ****************************************************************************************************
            

            Regards,
            JIRA Agile team

            Michael Tokar added a comment - This issue should now be resolved in JIRA Agile 6.6.13. What we have done to fix this is to put in place additional checks to figure out what collation is currently being used by the underlying database. If those checks tell us that the collation is non-standard, we apply additional sorting functions to our SQL queries when performing rank operations, to ensure that our ranking algorithm still performs as expected. In this situation however, performance will be impacted, as applying these additional sorting functions slows down the operation of the database. There is also a message output in the logs when we detect this case: **************************************************************************************************** The database collation is set incorrectly for JIRA Agile. This can cause JIRA Agile to run slower. If you are using PostgreSQL 9.0 or earlier, this may also cause ranking to fail. Please refer to the following documentation on the database configuration: https: //confluence.atlassian.com/display/DOC/Database+Configuration If the problems persist, please refer to this KB on how to fix the database: https: //confluence.atlassian.com/x/PAQWK **************************************************************************************************** Regards, JIRA Agile team

            As stated in JRA-39970 it seems as though Jira will in the future require a specific collation. This is a poor solution to the problem in my opinion, as it excludes people who need a different collation.

            Oskar Hanberg added a comment - As stated in JRA-39970 it seems as though Jira will in the future require a specific collation. This is a poor solution to the problem in my opinion, as it excludes people who need a different collation.

            MattS added a comment -

            Please confirm that this issue is not a problem with the default language setting for JIRA

            MattS added a comment - Please confirm that this issue is not a problem with the default language setting for JIRA

            Thanks for your comments on this issue so far. We will most likely tackle this issue as part of JRA-39970 and GHS-11162.

            Regards,
            JIRA Agile Team

            Michael Tokar added a comment - Thanks for your comments on this issue so far. We will most likely tackle this issue as part of JRA-39970 and GHS-11162 . Regards, JIRA Agile Team

            vkharisma added a comment -

            Thanks Erik, we modified the bug summary to be more generic and also updated the diagnosis section.

            vkharisma added a comment - Thanks Erik, we modified the bug summary to be more generic and also updated the diagnosis section.

              Unassigned Unassigned
              jtye Joe Wai Tye (Inactive)
              Affected customers:
              7 This affects my team
              Watchers:
              22 Start watching this issue

                Created:
                Updated:
                Resolved: