@mlassau the comment at https://jira.atlassian.com/browse/JRA-38598?focusedCommentId=653398&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-653398 showed a significant reduction in the reindexing time based on changing the number of threads. Perhaps the DB read actually is a bottleneck

Good to know about jira-config.properties. We will test for performance differences next time we upgrade. This issue really is resolved.

For background re-indexing, those of us with high up-time requirements have been depending on the JIRA Script Runner plugin which has allowed background reindexing of arbitrary sets of JIRA tickets for years. I will inquire with that author about the behavior with respect to the new property setting.

Thank you for updating this issue promptly!

This issue is asking for the number of threads to be tunable...

We added a property called "jira.index.issue.threads" to jira-config.properties that affects foreground reindex.

We intentionally made the background reindex run in a single thread because, well, its running the background, and we want to limit the performance degradation for other parallel operations.

... even when Background re-indexing.

Really? This issue was raised before Background reindex was a reality, so I am going to have to disagree with you

Modern servers have far more than 10 cores, so why limit the reindex to 10 threads?

Like I said in a previous comment, the Lucene writer is single-threaded (see eg http://lucene.472066.n3.nabble.com/IndexWriter-multithreaded-td569696.html).
The 10 threads are only reading from the DB to feed the single Lucene writer thread, so I doubt if increasing that number is going to help anyway.

daved: The reindex choice is presented as:

There is no "Are you sure?" double-clutch, but it does make it reasonably clear that the operation may take a while and that the instance will be unavailable during that time. Reindexing itself is also much faster in more recent versions of JIRA.

Hi - We've just been bitten by this "foreground reindex" issue for the 3rd time: the link to kick off a reindex in v4.4.5 starts the reindex with no "Are You Sure?" message. New Jira admins see the message saying that a reindex is needed, and they click the link. (Now, why we hire people who will press a button on a major system without knowing what it will do is another question...)

But anyway, this has happened to us at least 3 times over the past 5 or 6 years, and it takes Jira offline for a couple of hours in the middle of the day! Not cancellable! MAJOR pain in the neck! The whole thing could easily be mitigated if the process first presented a message that says, "This will take Jira offline for the duration, and may take hours. Are you sure you want to do this?"

In Mark Lassau's comment from 04/Jul/12, it says, "The admin has the choice of the two options when they start the reindex and the background re-index is cancellable."

I hope the design includes some kind of "Are you sure?" message for the foreground option.

Hi,

Great thanks for the background reindex !!!

Regards,
Sven.

Also, FWIW, our current re-index rate is in the 250 to 300 issues per second range for foreground re-indexing. I don't have good numbers for background re-index performance with the Script Runner plugin.

I can move this request to a separate issue, but operationally the difference between reindexing the whole instance, which could take hours, and reindexing just a few hundred or few thousand issues, which would take minutes, means the difference between something that can be done during business hours vs. something that may have to be done on a weekend in order to avoid prolonged performance issues and/or unavailability of data.

When tickets are imported from other issue tracking systems, especially if we're massaging the data in the database after import

This makes sense if you are manually massaging issues in the DB.

Changing the value of a custom field across all tickets

In this case you would want to do a full re-index (or at least for all projects that use the custom field which suggests more than a few hundred issues)

Changing the type of a custom field (select-list to multi-select list, say)

In this case you would want to do a full re-index (or at least for all projects that use the custom field which suggests more than a few hundred issues)

Recovering from a JIRA crash where the index has become corrupted for a handful of tickets

Do you actually know which issues the index is corrupted for so you can limit the re-index? How?
It seems to me that for this case you would be better off to do a full background re-index?
Perhaps you want a "selective" re-index to fix the high priority issues that were spotted ASAP, followed by a full re-index to clean up the ones you haven't noticed?

Anyway, at this stage I should suggest you open a separate improvement request where we can gather this information...

Re: business case

We use on-line re-indexing for the following:

• When tickets are imported from other issue tracking systems, especially if we're massaging the data in the database after import
• Changing the value of a custom field across all tickets
• Changing the type of a custom field (select-list to multi-select list, say)
• Recovering from a JIRA crash where the index has become corrupted for a handful of tickets

In these cases, we're not changing the meaning of the data in a way that would motivate us to want the Updated Date to change.

Re: configurable number of index threads

Looks great! We will test and let you know if we see performance improvements or not.

Anyway, back on track to the original issue posted here...

We have added a property called "jira.index.issue.threads" to jpm.xml for v5.2

This will effect the "foreground" reindex only (the old one, not the background reindex I mentioned above).
We have not experimented with changing this value, but our guess is that it will probably will not help.
Ultimately, the multiple threads are just gathering data to send to Lucene for indexing, but the Lucene Index writer is single-threaded by design and likely to be the bottleneck.

Greg, could you supply some details of why you need this?
I don't get the business case.

As a "workaround" you could do a trivial edit on said issues?

Perhaps "handful" was the wrong word, but we would consider hundreds to be a small number is issues to reindex, which we commonly do. Please consider making a partial re-index (based on a JQL query, say) part of this functionality.

Also, any time we affect the Updated Date on an issue unnecessarily there is significant disruption to our user's workflows, so trivial edits are a bad workaround.

gbrauer1 Actually the background reindex does not "build a new index and cut over" - rather it re-indexes the issues in place one at a time.
There is no option to reindex only some issues.

Often there are only a handful of issues...

Every time you update an issue (or add/delete a comment) we re-index that issue in place.
As a "workaround" you could do a trivial edit on said issues?

Mark, this sounds great.

Will there be any option to background reindex only some issues, or does the "build a new index and cut over" strategy preclude that? Often there are only a handful of issues that we know need reindexing and the Script Runner plugin allows us to take care of those in a few seconds instead of an hour.

Hi Mark,

Good to know, this new feature will improve a lot our service to the user.
I'm belonging to a JIRA production team (10 000 users, 6 JIRAs instances), this such of feature are very important for user.
continue in this way !!!

Regards,
Sven.

In 5.2 we are planning to introduce a "background reindex" feature.

Currently a re-index will lock users out of JIRA, throw away the old index, and then rebuild the full index from scratch before we let users in again.

The background re-index assumes that the indexed data is mostly correct, but there has been a configuration change that requires a small change in the index.
It doesn't lock users out, and updates the existing index a few issues at a time in just 2 threads.
Performance testing shows that it takes about twice as long to run in total as the "foreground reindex", and there is a performance hit of about 20% on some pages.
However, users can still use JIRA in the meantime.

The admin has the choice of the two options when they start the reindex and the background re-index is cancellable.

This feature will be available for testing in the first 5.2 milestone which will be available in 2 weeks or so.

We've got an instance with just over 400,000 issues and it currently takes 23 1/2 minutes to index.
However :

• The CPU load is never more than 50%
• IO Wait is at 0% throughout the index
• And we don't experience any Full GCs (using ParallelOldGC with 12G Heap) during the index

The system has a single Xeon X5670 with 6 cores (12 threads), 48GB RAM, and we're running Java 6 Update 26 with MySQL Connector/J 5.1.18 on CentOS 5.5
We found that MySQL query cache was doubling the index time because of query cache mutex waits (we're using MySQL 5.1.59, so I can't speak for 5.5.x), so we've disabled it completely before indexing to obtain the time above. We then re-enable the query cache after indexing is complete.
We also found that the Salesforce plugin we use was dialing out to SFDC as part of the index process and adding another 5 minutes to the indexing time.

It makes sense that for large instances like our own and the instances in previous comments should have the ability to increase the number of threads dedicated to the indexing process.
In our case, we could probably increase it to 15-20 threads to saturate the system between Jira and the local MySQL server it uses (assuming a fully warmed buffer pool)

Can Atlassian even comment on the feasibility of introducing such an option?

We also have an instance that is 600K+ issues. We would like to get the time to under 5 minutes.

Currently Jira allocate 10 threads for the re-index.
We have a large instance 600 000 issues that take a while to re-index (100 min).

Could you please implement a functionnality to increase the capacity to re-index Jira.

Regards,
Sven.