-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
34
-
Severity 2 - Major
-
10
-
Issue Summary
Customer's use case:
Discovery has 1 scan settings configured with a single IP - a vCenter.
The option "Try VM Guest scan" is set to True.
The vCenter hosts many VMs, some of these are also hosting VMs on them.
The Scan Results Zip contains ~630 Hosts, and is ~46.5MB (compressed) / 216MB (uncompressed).
The file takes a long time to load into ScanViewer.exe - but it eventually loads - showing all the Hostinfo Objects.
Steps to Reproduce
- Use a Collector to Export this file to a Cloud Instance (using a valid Token)
- Upload the file manually to Cloud, and execute a sync
- Try importing the same file into Insight On-Premise
Expected Results
The import is executed, and may take a long time to process. Any error will be logged / showing.
Actual Results
On Cloud: File uploads successfully, no visible / logged errors.
1. UI : Process Results tab - shows a process is in progress, then it stops showing this process, and no trace of the import was executed.
2. If Manually uploaded and triggered - Progress starts to show, but never advances. Refreshing the page will remove the Progress window. No trace for the running import (neither in the Process results tab)
3. On Premise gets an OOM - Java heap memory (~9GB of JVM Memory max enabled on testing)
The below exception is thrown in the atlassian-jira.log file:
2022-01-25 18:49:08,837-0500 insight-object-updater-0-thread WARN anonymous [i.r.i.index.model.ObjectIndexImpl] Unable to reindex object null Java heap space
2022-01-25 18:49:08,836-0500 insight-value-updater-0-thread ERROR [c.r.j.p.i.s.c.d.d.sql.batch.ObjectAttributeValueBatchController] Unable to flush database values. Contact support
java.lang.OutOfMemoryError: Java heap space
2022-01-25 18:49:12,080-0500 pool-20-thread-1 WARN yinon 951x184x12 3l7o8m 0:0:0:0:0:0:0:1 /rest/insight/1.0/progress/result/objectschema/6 [atlassian-diagnostics] 2022-01-25T23:49:11.402Z Component 'Database' alerted 'Database connection leaked' (details: {"activeConnections":15,"idleConnections":15,"maxConnections":30,"connectionAcquiredTimestampInMillis":1643140633645}, trigger: {"pluginKey": "not-detected"})
The Insight-import.log shows:
2022-01-25 18:49:52,078 [insight-InsightThreadGroup-worker-thread-8] | Execution exception when reading data java.lang.OutOfMemoryError: Java heap space 2022-01-25 18:49:58,719 [insight-InsightThreadGroup-worker-thread-8] | Execution exception when reading data java.lang.OutOfMemoryError: Java heap space 2022-01-25 18:49:58,720 [insight-InsightThreadGroup-worker-thread-8] | di2co>>> Fetched all data locators from external source in 14037573 ms 2022-01-25 18:49:58,754 [insight-InsightThreadGroup-worker-thread-8] | di2co>>> Mapped all data to internal objects in 33 ms 2022-01-25 18:49:58,758 [insight-InsightThreadGroup-worker-thread-8] | di2co>>> Decided actions for all objects in 4 ms 2022-01-25 18:49:58,758 [insight-InsightThreadGroup-worker-thread-8] | di2co>>> Objects to import 0 2022-01-25 18:49:58,758 [insight-InsightThreadGroup-worker-thread-8] | di2co>>> Finished import with id 11 of type insight-discovery-import at time 2022-01-25T18:49:58.758. Import took 14037670 ms 2022-01-25 18:49:58,766 [insight-InsightThreadGroup-worker-thread-8] | com.riadalabs.jira.plugins.insight.services.imports.common.ImportResult@2539289f[ status=FINISHED started=Tue Jan 25 14:56:01 EST 2022 ended=Tue Jan 25 18:49:58 EST 2022 executedType=MANUAL executedAsUserKey=JIRAUSER10000
Workaround
Break your Scan into separate Scan Settings, with a few IP ranges in each Scan Settings to cover the full range of IPs.
Be sure to adjust the Missing Objects Threshold in each Object Type Map (import configuration) to at least n+1 (n=the number of Scan Settings configured)
- is related to
-
-
- Gathering Impact
-
- was cloned as
-
-
- Closed
-
- mentioned in
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Loading...
-
[JSDCLOUD-11162] Insight Discovery import of a large Scan Results zip does not work
Ran some tests here, by importing Discovery files:
| Hosts | Devices | Total # of objects created | Zip size | Actual file size | Duration | Cloud Result | 2nd run duration | 2nd run # of objects added |
|---|---|---|---|---|---|---|---|---|
| 500 | 100 | *83,497 | 10.5M | 41M | 70m |  |
n/a | n/a |
| 1000 | 100 | **47,546 | 22M | 86M | 56m | |
***n/a | 120,657 |
| 7000 | 2000 | n/a | 265M | 895M | n/a |  |
* The first import had ALL Object Type Maps enabled.
** Subsequent imports - with default configuration; Virtual Guest, Application, Application Service, Patches, Users & Group object type maps were disabled for the initial run, then enabled for the second run (same file uploaded)
***2nd run for 1000 Hosts did not register any process result
The 3rd import (7000 Hosts, with default settings) did not complete:
The file was uploaded from Discovery correctly, and was deleted from the Discovery/scans directory.
The import was triggered - and progress bar was showing.
The import failed after 5-7m with an error banner:
Failure to fetch progress of di2coPlease check your connection or try again at a later time. If problem persist, please contact support.
This bug makes it really complicated: if you split the scan settings into several ranges, but still write the data into the same objects in the cloud, you never get accurate status infos on the actual objects. If you set the missing object threshold too low, objects will have wrong status or even be removed, if you set the threshold to high, of have a lot of out-dated objects.
When combining Assets Discovery Service with Assets Discovery Collector (-> Assets Discovery Agents), there is no good solution. It is always compromises...
Thank you vm Yinon.. i'm sorry but i'm not completely understand, for example, if i have 3 file a, b, c i put the threshold to 4 and:
- after the first import i have already loaded
- after the second import of A, all the data loaded with B and C, it will be updated as "not found"?
Regards
ema
Hi Ema.
In your example; if you have 10 scan settings, set Missing Objects Threshold to 11 or more (12, 14, or even 20).
So 10 scan results will import ALL Hosts. And in the second run - if an object is missing from ALL 10 Scan results - it will be acted upon as a Missing object once when the threshold is reached.
The headroom (setting threshold >10) is there in case the files are not uploading in the same order... (e.g. if Scan one is of the second run is not uploaded because there is still an import in progress...) This is not ideal, and works differently than available configuration on Data Center (see JSDCLOUD-11228 ) - and since the Cloud import is "passive" (triggered by Discovery / Collector) using the synchronization counter as threshold - we need to find a balance so that really missing objects will be acted upon as missing objects (trying to keep a "tight"/low threshold number to detect Missing objects ASAP) and Hosts / objects not "fluctuating" by being falsely tagged as Missing / Running continuously (we do that by increasing the threshold)
Kind regards,
Yinon
Atlassian | Insight Support
Hi All,
what about the workaround? "Be sure to adjust the Missing Objects Threshold in each Object Type Map (import configuration) to at least n+1 (n=the number of Scan Settings configured)"
could you please better clarify?
the "missing object" will be setted or not? and.. what about, if we have:
- round one divided in 10 scan --> ok
- round two divided in 10 scan --> what will happen if an object is really missing? how will manage it?
txs
ema
Additional tests, a single XML file (Only Hosts from Import 3 above) in the Scan result zip:
Running the same import again (UI) or using discovery.exe -z results in the "Failure to fetch progress" error, and the file is uploading but not importing...