Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-68653

Asynchronous cache replication queue - leaking file descriptor when queue file corrupted

    XMLWordPrintable

Details

    Description

      Problem

      If a cache replication queue is corrupted when a node is shutting down, then on the next node start Jira will try to open this queue file every time it is required (a cache replication message is being send to another node on this particular channel = file). If the existing file is corrupted it fails with the following error:

      ERROR      [c.a.j.c.distribution.localq.LocalQCacheManager] Error when creating cache replication queue for node: [node_name]. This node will be inconsistent. Error: File is corrupt; length stored in header is 0.

      This results in:

      • cache replication message not being delivered
      • leak of file descriptor
        • Jira hits into "Too many open files" error. Reviewing lsof output points to many localq entries.

      Desired Jira behaviour

      If the file is corrupted backup this file (copy with corrupted_ prefix) and create a new file.

      Workaround

      Delete the corrupted queue file. Step to identify the corrupted file can be found following comment-1917799
      It should not be necessary to shut down this node. It should recreate this queue file automatically. 
       

      Attachments

        Issue Links

          Activity

            People

              mswinarski Maciej Swinarski
              mswinarski Maciej Swinarski
              Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: