If a cache replication queue is corrupted when a node is shutting down, then on the next node start Jira will try to open this queue file every time it is required (a cache replication message is being send to another node on this particular channel = file). If the existing file is corrupted it fails with the following error:
This results in:
- cache replication message not being delivered
- leak of file descriptor
- Jira hits into "Too many open files" error. Reviewing lsof output points to many localq entries.
If the file is corrupted backup this file (copy with corrupted_ prefix) and create a new file.
Delete the corrupted queue file. Step to identify the corrupted file can be found following comment-1917799
It should not be necessary to shut down this node. It should recreate this queue file automatically.