Problem

On large instance (ie. 5000+ projects), deleting applink can cause system slowness. 

Steps to reproduce

1. Jira with 5000+ projects 
2. Create a dummy applink 
3. Delete the applink

Result

Jira triggers 5000+ database calls just to remove the applink properties, subsequently halting other operations until the applink is removed. In this case, can take about 5-10minutes.

Diagnostic

In atlassian-jira-sql.log you can see the following line /rest/applinks/3.0/applinks/<applink-id> iterating through all jira_project.linked.entities, which can be large in Enterprise.

2018-03-12 04:52:31,780 http-nio-8080-exec-1 DEBUG 1572322 292x422x1 togqcz 192.168.32.8 /rest/applinks/3.0/applinks/09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad [c.a.applinks.core.DefaultApplicationLinkService] Deleting application link id [09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad]
2018-03-12 04:52:31,780 http-nio-8080-exec-1 DEBUG 1572322 292x422x1 togqcz 192.168.32.8 /rest/applinks/3.0/applinks/09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad [c.a.applinks.core.DefaultApplicationLinkService$DefaultEntityLinkService] Deleting Entity Links for Application Link [09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad]
2018-03-12 04:52:31,832 http-nio-8080-exec-1 DEBUG 1572322 292x422x1 togqcz 192.168.32.8 /rest/applinks/3.0/applinks/09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad [c.a.applinks.core.DefaultApplicationLinkService$DefaultEntityLinkService] Setting stored entity links for [DEVOPS] [jira.project] as [[]]
2018-03-12 04:52:31,833 http-nio-8080-exec-1 DEBUG 1572322 292x422x1 togqcz 192.168.32.8 /rest/applinks/3.0/applinks/09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad [c.a.a.core.property.SalPropertySet] Putting property [linked.entities] as namespace [applinks.local.DEVOPS.jira_project.linked.entities] with value [[]]
2018-03-12 04:52:31,833 http-nio-8080-exec-1 DEBUG 1572322 292x422x1 togqcz 192.168.32.8 /rest/applinks/3.0/applinks/09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad [c.a.a.core.property.HashingLongPropertyKeysPluginSettings] Putting key [applinks.local.DEVOPS.jira_project.linked.entities] as hashKey [applinks.local.DEVOPS.jira_project.linked.entities] with value [[]]
2018-03-12 04:52:31,951 http-nio-8080-exec-1 DEBUG 1572322 292x422x1 togqcz 192.168.32.8 /rest/applinks/3.0/applinks/09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad [c.a.applinks.core.DefaultApplicationLinkService$DefaultEntityLinkService] Setting stored entity links for [DQIM] [jira.project] as [[]]
2018-03-12 04:52:31,952 http-nio-8080-exec-1 DEBUG 1572322 292x422x1 togqcz 192.168.32.8 /rest/applinks/3.0/applinks/09fd3e1f-a6a1-359f-9dc9-2efd88b1f5ad [c.a.a.core.property.SalPropertySet] Putting property [linked.entities] as namespace [applinks.local.DQIM.jira_project.linked.entities] with value [[]]

Root cause. 

When deleting applink, the following happen

1. com.atlassian.applinks.core.DefaultApplicationLinkService#deleteApplicationLink gets called
2. It calls com.atlassian.applinks.core.DefaultApplicationLinkService#deleteEntityLinksFor
3. This method performs operations on the linked entities one-by-one in a loop (at least one linked entity exists for every project on the instance)
4. Indirectly, some locks are being held during the operations, (e.g. by accessing some collections shared with other threads)
5. Each iteration performs a query to the database (while holding the lock)
6. Our client has over 5000 properties and each query, according to the logs, takes more than 50ms to complete. This, coupled with the locking slows entire Jira instance down.

The loop inside deleteEntityLinksFor should be optimized to perform the deletes in bulk with only one query, as the 5000 roundtrips to the database are the bottleneck here. 

Workaround. 

None. Avoid deleting Applink during peak hour. 

Timeout error and Config error when creating an application link

• Since it takes some time to delete the application link, even if the UI indicated the application link is deleted, Jira's log is still deleting the entity links in the background.

  2019-04-19 14:48:31,802 http-nio-8192-exec-22 DEBUG sasa 886x797x1 1ni6qxl 0:0:0:0:0:0:0:1 /rest/applinks/3.0/applinks/a0e7c7c2-8633-3002-8379-538907da08a0 [c.a.applinks.core.DefaultApplicationLinkService$DefaultEntityLinkService] Setting stored entity links for [YENE] [jira.project] as [[]]
  2019-04-19 14:48:31,802 http-nio-8192-exec-22 DEBUG sasa 886x797x1 1ni6qxl 0:0:0:0:0:0:0:1 /rest/applinks/3.0/applinks/a0e7c7c2-8633-3002-8379-538907da08a0 [c.a.a.core.property.SalPropertySet] Putting property [linked.entities] as namespace [applinks.local.YENE.jira_project.linked.entities] with value [[]]
  

• If a new application link is created during this time, timeout error and config error will be returned in the UI.

Workaround

• Suggest waiting until the deletion completely finished before creating a new application link.
• From our testing, we suggest waiting for 10 minutes at least for Jira instance with 3000 projects.