Scheduled index snapshot silently fails if parent directory doesn't exist

Problem

Index snapshot silently fails if parent directory doesn't exist

Environment

Linux
Tested in Jira 9.4.9 and 9.10.1

Steps to Reproduce

1. Set up JDC and enable scheduled index snapshot
2. Verify the directory of the snapshot. In my case it is

   /Users/shared-home/atlassian-jira-software-9.10.1-standalone/caches/indexesV2/snapshots

3. Remove the parent directory: indexesV2
4. Check if index snapshot is successfully created

Expected Results

1. Jira will automatically create indexesV2/snapshots directory if it doesn't exist and snapshot should successfully be created/saved in the /Users/shared-home/atlassian-jira-software-9.10.1-standalone/caches/indexesV2/snapshots OR
2. Throw an error in the UI for admin to take action if there is failure in creating the snapshot

Actual Results

1. Jira doesn't create the necessary directory and the snapshot will not be created/saved
2. No errors in the UI about this failure. It's just a silent failure which nobody will notice until snapshots are actually required
3. Following is thrown in atlassian-jira.log

   2023-08-30 12:28:00,004+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.DefaultIndexSnapshotOperator] Attempting to create snapshot, attempt number: 1
   2023-08-30 12:28:00,005+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.issue.index.DefaultIndexManager] Start to check for index consistency.
   2023-08-30 12:28:00,006+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.issue.index.DefaultIndexManager] Index consistency check finished. indexConsistent: true; timeToCheckIndexConsistency: 0 ms
   2023-08-30 12:28:00,007+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.IndexUtils] Committing writers to disk before taking a snapshot took PT0.000119757S
   2023-08-30 12:28:00,007+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.IndexUtils] Creating index snapshot of type: SNAPPY in local temporary directory: /Users/jira/atlassian-jira-software-9.10.1-standalone/temp/snapshot1693391280007/IndexSnapshot_10102_230830-122800.tar.sz
   2023-08-30 12:28:00,007+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.util.compression.ArchiveUtils] Creating archive of type: SNAPPY in path: /Users/jira/atlassian-jira-software-9.10.1-standalone/temp/snapshot1693391280007/IndexSnapshot_10102_230830-122800.tar.sz
   2023-08-30 12:28:00,008+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.issue.index.DefaultIssueIndexer] IssueIndexer starts adding 4 indexes to backup from: [issues, worklogs, comments, changes]
   2023-08-30 12:28:00,008+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Creating 4 commit points for Lucene indexes
   2023-08-30 12:28:00,009+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Adding 15 files from issues to backup...
   2023-08-30 12:28:00,015+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Done adding 15 files from issues to backup, time: 6 ms
   2023-08-30 12:28:00,015+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Adding 1 files from worklogs to backup...
   2023-08-30 12:28:00,016+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Done adding 1 files from worklogs to backup, time: 0 ms
   2023-08-30 12:28:00,016+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Adding 15 files from comments to backup...
   2023-08-30 12:28:00,022+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Done adding 15 files from comments to backup, time: 6 ms
   2023-08-30 12:28:00,023+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Adding 13 files from changes to backup...
   2023-08-30 12:28:00,029+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Done adding 13 files from changes to backup, time: 6 ms
   2023-08-30 12:28:00,029+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Releasing 4 commit points for Lucene indexes
   2023-08-30 12:28:00,029+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.issue.index.DefaultIssueIndexer] IssueIndexer done adding 4 indexes to backup from: [issues, worklogs, comments, changes]
   2023-08-30 12:28:00,029+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.sharing.index.DefaultSharedEntityIndexManager] SharedEntityIndexer starts adding 2 indexes to backup
   2023-08-30 12:28:00,030+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Creating 1 commit points for Lucene indexes
   2023-08-30 12:28:00,030+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Adding 13 files from entities/searchrequest to backup...
   2023-08-30 12:28:00,036+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Done adding 13 files from entities/searchrequest to backup, time: 5 ms
   2023-08-30 12:28:00,036+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Releasing 1 commit points for Lucene indexes
   2023-08-30 12:28:00,036+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Creating 1 commit points for Lucene indexes
   2023-08-30 12:28:00,036+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Adding 13 files from entities/portalpage to backup...
   2023-08-30 12:28:00,041+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Done adding 13 files from entities/portalpage to backup, time: 4 ms
   2023-08-30 12:28:00,041+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.SnapshotDeletionPolicyContributionStrategy] Releasing 1 commit points for Lucene indexes
   2023-08-30 12:28:00,041+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.sharing.index.DefaultSharedEntityIndexManager] SharedEntityIndexer done adding 2 indexes to backup
   2023-08-30 12:28:00,042+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.IndexUtils] Done creating index snapshot: IndexSnapshot_10102_230830-122800.tar.sz, number of index files processed: 70
   2023-08-30 12:28:00,042+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.IndexUtils] Compressing files took 34 ms
   2023-08-30 12:28:00,042+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.IndexUtils] Copying index snapshot from: /Users/jira/atlassian-jira-software-9.10.1-standalone/temp/snapshot1693391280007/IndexSnapshot_10102_230830-122800.tar.sz to remote shared directory: /Users/shared-home/atlassian-jira-software-9.10.1-standalone/caches/indexesV2/snapshots/IndexSnapshot_10102_230830-122800.tar.sz
   2023-08-30 12:28:00,042+0200 Caesium-2-2 ERROR anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.IndexUtils] Error when copying index snapshot from: /Users/jira/atlassian-jira-software-9.10.1-standalone/temp/snapshot1693391280007/IndexSnapshot_10102_230830-122800.tar.sz to remote shared directory: /Users/shared-home/atlassian-jira-software-9.10.1-standalone/caches/indexesV2/snapshots/IndexSnapshot_10102_230830-122800.tar.sz! WARNING - Without up-to-date index snapshots regularly saved to shared home time to start/restart nodes in the cluster might significantly increase!  Other nodes will not be able to use recent index snapshots and might need to perform a full reindex on start. Make sure shared home is accessible for this node and has sufficient amount of space to accept new index snapshots! Current shapshot size: 53 KB
   java.nio.file.NoSuchFileException: /Users/shared-home/atlassian-jira-software-9.10.1-standalone/caches/indexesV2/snapshots/IndexSnapshot_10102_230830-122800.tar.sz
   	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
   	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
   	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
   	at java.base/sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:248)
   	at java.base/sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:603)
   	at java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258)
   	at java.base/java.nio.file.Files.copy(Files.java:1295)
   	at com.atlassian.jira.index.ha.IndexUtils.copyFiles(IndexUtils.java:243)
   	at com.atlassian.jira.index.ha.IndexUtils.doPerformBackupOperations(IndexUtils.java:187)
   	at com.atlassian.jira.index.ha.IndexUtils.performBackupOperations(IndexUtils.java:144)
   	at com.atlassian.jira.index.ha.DefaultIndexSnapshotOperator.performIndexSnapshotBackupAndCleanup(DefaultIndexSnapshotOperator.java:188)
   	at com.atlassian.jira.index.ha.DefaultIndexSnapshotOperator.tryToCreateSnapshotWhile(DefaultIndexSnapshotOperator.java:119)
   	at com.atlassian.jira.index.ha.DefaultIndexSnapshotOperator.tryCreateSnapshot(DefaultIndexSnapshotOperator.java:109)
   	at com.atlassian.jira.index.ha.IndexSnapshotService.run(IndexSnapshotService.java:39)
   	at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:68)
   	at com.atlassian.jira.service.ServiceRunner.runService(ServiceRunner.java:62)
   	at com.atlassian.jira.service.ServiceRunner.runServiceId(ServiceRunner.java:44)
   	at com.atlassian.jira.service.ServiceRunner.runJob(ServiceRunner.java:32)
   	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
   	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
   	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
   	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
   	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
   	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
   	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
   	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
   	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
   	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
   	at java.base/java.lang.Thread.run(Thread.java:834)
   2023-08-30 12:28:00,044+0200 Caesium-2-2 INFO anonymous    JIRA Index Snapshot Service [c.a.j.index.ha.DefaultIndexSnapshotOperator] On attempt number 1 snapshot creation status was ERROR. Finishing.

Workaround

Manually ensure that the snapshot directory exists in your environment

Notes

/Users/shared-home/atlassian-jira-software-9.10.1-standalone/caches/indexesV2/snapshots

• If the main directory itself (snapshots) doesn't exist, Jira will automatically create it and it will work fine
• It's only a problem is the parent directory (indexesV2) doesn't exist