Issue Summary

Assets Azure Integration import takes many hours to complete if there are a large number of Users and Groups objects.

The impacted customers may have more than 20,000 objects, although the actual number could vary, potentially being more or less.

This Bug is created from a current FR:
JSDSERVER-12030 - Import from Azure into Insight (Assets) is taking a very long time to complete

This is reproducible on Data Center: yes

Steps to Reproduce

1. Set up Assets Azure Integration
   • It is necessary to have a substantial Azure instance with more than 20,000 users and groups for the import process.
2. Ensure to enable 'Users' and 'Groups' Object Type mapping within the import
3. Synchronize the import

Expected Results

Import should not take so long

Actual Results

For a customer issuer, There were 31,454 groups to sync from Azure, debug stats show that the majority of the time (37 hours avg) is spent reading data in Azure, and not anything such as writing the data that has been fetched to Insight (Assets).

The server CPU and Memory utilization are also very low which suggests that this is not a performance problem within Jira or Insight (Assets).

• 33 avg minutes for object type user
• 37 hours only for the object type group

Feedback from Development:

This group import is slow because as mentioned above there are 31454 group to sync. When loading groups, we call external Azure API to load page by page and we load 25 group at a time. This means there must be 1259 API calls to load all the groups from Azure to sync. Call 1259 API calls to sync from azure should not take 39 hours. However, for each group `insight-azure-integration` is making an extra API call to get the list of `members` for each group. This results in at least extra 31454 API calls to sync the members of each group. This leads to minimum total 31454 + 1259 = 32173 API requests to Azure which explains why it was so slow.

In addition, 32173 API calls is just an ideal number of minimum requests that will be created, in reality, the number of requests is larger, and we can't guess as it depends on the real situation of customer Azure data.

For example, If there are 1000 groups in 31454 groups which those 1000 groups have 250 members each. When loading members for each group, we are also fetching page by page and 25 per page. This means it's will be taking 250 / 25 * 1000 = 10000 requests to get members of those 1000 groups. Then the final total number of requests to Azure needed is 1259 + 10000 + (31454 - 1000) = 41713 API requests to Azure needed to sync everything.

However, even with the minimum number of requests to be made to azure (32173 requests), it's already massive enough to be slow in progressing while the true number may even be much higher.

You may ask why the server looks to be IDLE, CPU and memory chart looks fine. It is because calling requests to Azure API when syncing does not take much memory or CPU or disk space. It is just taking time.

Thread Dumps

One thread is doing the GET against azure, seems to be single threaded. Groups is taking ages to completed

- insight-InsightImportThreadGroup-worker-thread-8
com.microsoft.graph.auth.confidentialClient.ClientCredentialProvider.getAccessTokenNewRequest(ClientCredentialProvider.java:102)
com.microsoft.graph.auth.confidentialClient.ClientCredentialProvider.getAcccessToken(ClientCredentialProvider.java:67)
com.microsoft.graph.auth.confidentialClient.ClientCredentialProvider.authenticateRequest(ClientCredentialProvider.java:49)
com.microsoft.graph.http.CoreHttpProvider.sendRequestInternal(CoreHttpProvider.java:395)
com.microsoft.graph.http.CoreHttpProvider.send(CoreHttpProvider.java:220)
com.microsoft.graph.http.CoreHttpProvider.send(CoreHttpProvider.java:200)
com.microsoft.graph.http.BaseCollectionRequest.send(BaseCollectionRequest.java:92)
com.microsoft.graph.requests.extensions.GroupCollectionRequest.get(GroupCollectionRequest.java:58)
com.riadalabs.jira.plugins.azure.imports.manager.impl.GroupService.get(GroupService.java:173)

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available