Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-51950

Indexer does not add large number of attachments to Journal Table

    XMLWordPrintable

Details

    Description

      Summary

      When a page is updated and needs to be indexed, the indexer does not add the attachments, comments and permissions into the journal table. This will overload the hibernate session, where it will pick up items from the journal table. These items will introduce more work on the indexer than expected, where it will need to process all the attachments, comments and permissions of the current page.
      From the user perspective, it will look like the indexer stalls and slow at processing a small number of tasks

      Steps to Reproduce

      1. Create a new page
      2. Add multiple attachments to the page (preferably large attachments)
      3. Edit the page again and save
      4. Enable DEBUG logging for the package
        com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager
        

      Expected Results

      The indexer will show that it is processing the page along with all the attachments, comments and permission

      Actual Results

      The indexer will show that it is processing a single page only while it is actually processing page, attachments, comment and permissions.

      Large attachments can affect the processing time as well. Below is the log snippet of this behavior:

      2017-03-21 11:15:03,887 DEBUG [scheduler_Worker-4] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 291 items in 43881 milliseconds
      2017-03-21 11:14:20,006 DEBUG [scheduler_Worker-4] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      2017-03-21 11:14:16,240 DEBUG [scheduler_Worker-1] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 2338 items in 276234 milliseconds
      2017-03-21 11:09:40,005 DEBUG [scheduler_Worker-1] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      2017-03-21 11:09:36,402 DEBUG [scheduler_Worker-2] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 1527 items in 466396 milliseconds
      2017-03-21 11:01:50,006 DEBUG [scheduler_Worker-2] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      2017-03-21 11:01:47,935 DEBUG [scheduler_Worker-1] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 1778 items in 667844 milliseconds
      2017-03-21 10:50:40,091 DEBUG [scheduler_Worker-1] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      2017-03-21 10:50:36,342 DEBUG [scheduler_Worker-6] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 869 items in 26337 milliseconds
      2017-03-21 10:50:10,004 DEBUG [scheduler_Worker-6] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      ....
      2017-03-21 10:48:02,290 DEBUG [scheduler_Worker-4] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 6 items in 2285 milliseconds
      2017-03-21 10:48:00,005 DEBUG [scheduler_Worker-4] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      ....
      2017-03-21 10:36:50,333 DEBUG [scheduler_Worker-4] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 4 items in 10325 milliseconds
      2017-03-21 10:36:40,007 DEBUG [scheduler_Worker-4] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      ....
      2017-03-21 10:34:58,947 DEBUG [scheduler_Worker-9] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue Flushed 157 items in 3942 milliseconds
      2017-03-21 10:34:55,005 DEBUG [scheduler_Worker-9] [confluence.search.lucene.DefaultConfluenceIndexManager] flushQueue flush requested
      

      Workaround

      The current workaround is to reduce the index batch size in order to avoid overloading the hibernate session.
      It could be as low as

      index.queue.batch.size=100
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              rslaiby Rudy Slaiby
              Votes:
              4 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: