Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-38598

JIRA reindexing is too slow - doesn't use existing CPUs

XMLWordPrintable

    • 72
    • 41
    • We collect Jira feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      2021-04 - Jira 8.x update

      We believe at this point this problem is fixed, there were a number of changes in the recent version of Jira that improved the performance of reindexing.

      Just to name a few:

      • Upgrade to the recent Lucene version 7.x
      • Optimization for the Custom Fields computation
      • Increased number of Jira index threads to 20.

      That being said, we don't think the CPU is bottle-neck anymore and it can be properly utilized by Jira. Examples that we have some far, show problems related to slow DB performance or slow computation of specific user's Custom Fields, those are not related to Jira directly.

      Initial Description

      We have a big JIRA instance on a really good hardware:

      • ~300k issues
      • 48GB RAM
      • 32 cores
      • SSD in RAID
      • PostgreSQL on localhsot

      Still, reindexing does not increase the load above 4-5 (out of 32!), database is not under high load nor the filesystem.

      Clearly this means that indexing is not properly using the resources in order to index rapidly.

      Even with this configuration, a foreground reindex takes about one hour which is too much for a live system, in fact is too much even for a background reindex.

      top - 11:57:30 up 50 days, 21:35,  2 users,  load average: 4.43, 4.02, 3.79
      Tasks: 413 total,   3 running, 410 sleeping,   0 stopped,   0 zombie
      Cpu(s): 14.3%us,  0.6%sy,  0.0%ni, 84.8%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
      Mem:  49415024k total, 48661932k used,   753092k free,   171160k buffers
      Swap:  2097148k total,   139576k used,  1957572k free, 21143064k cached
      
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
      18668 root      20   0 23.4g 3.0g  50m S  241  6.3  54145:22 java
      13694 jira      20   0 25.2g 5.5g 496m S  110 11.6 108:55.12 java
      14552 postgres  20   0 11.4g 896m 887m S   39  1.9   4:05.14 postgres
      12496 postgres  20   0 11.4g 897m 890m R   24  1.9   2:07.22 postgres
      14548 postgres  20   0 11.4g 947m 940m R   19  2.0   3:51.61 postgres
      15014 confluen  20   0 16.9g 1.7g 8352 S   13  3.7   1196:28 java
      14238 postgres  20   0 11.4g 1.0g 1.0g S    8  2.1   5:37.47 postgres
      12499 postgres  20   0 11.4g 855m 850m S    8  1.8   2:06.02 postgres
      30049 postgres  20   0 11.4g  78m  73m S    6  0.2   0:02.86 postgres
      17921 postgres  20   0 11.4g  73m  66m S    2  0.2   0:03.98 postgres
       4955 nginx     20   0 50924  17m 2168 S    1  0.0 107:41.21 nginx
      47354 postgres  20   0 11.4g 4344 2168 S    1  0.0   6:24.91 postgres                        
      

      Solution

      By default, Jira uses 10 Reindexing threads. You can tune number of jira index threads.

              dd46af19d4d5 Carlos Vigier (Inactive)
              73f0b2e75f82 Sorin Sbarnea (Citrix)
              Votes:
              84 Vote for this issue
              Watchers:
              100 Start watching this issue

                Created:
                Updated:
                Resolved: