Details
-
Bug
-
Resolution: Fixed
-
Medium
-
None
-
35
-
Severity 2 - Major
-
54
-
Description
Issue Summary
When using the Insight Discovery Import - multiple files may be exported from Discovery to Insight Cloud using the Discovery Import.
Here are three scenarios:
1. One Discovery Instance may have a few Scan Settings running one after the other.
2. A few Discovery instances may run at the same time, using the same Cloud Import Token in their Export to Cloud.
3. Several Discovery Instances are moving the scan results, to be picked up and exported by the Collector.
All three scenarios will result in:
One Scan Result file is being Exported/importing in Cloud, while another Scan Results file is attempting to export to the same Import Configuration - as it is using the exact same Token.
As the import engine is busy - it returns a 400 error, and the job is not being queued up. The failed to export files will remain in the Scans directory.
Steps to Reproduce
- Few options to test:
- Configure 2 scan settings, each with one IP address and Export to Cloud. Make sure the 1st scan setting will return a lot of information (using many Patterns) and set the second Scan Setting to gather only basic information (Hostinfo patterns) from the Localhost running discovery. - Set export to a Cloud instance.
- Configure 2 Discovery Instances, use the same Cloud Import Token in both, and execute a scan in each, time the scans so that one scan is finishing right after the other (e.g. 1st instance to have a long range of IPs, and the other to have only one... trigger the second instance to scan when the first is almost done...)
- gather a few Scan Results files, into the Scans folder, configure Export to Cloud with a Valid token, execute Discovery.exe -z to trigger the file transport
- gather a few Scan Results files, into the Import folder, configure a Collector to Export to Cloud with a Valid token, polling interval 1m, and install it as a service / trigger it manually.
- Check the logs (Discovery / Collector) and revise the Schema in Insight Cloud
Expected Results
All file are exported to Cloud, Scan/Import folder is empty, all the scanned info is imported and visible in Cloud.
Actual Results
One file is Exported to cloud, all other files are kept in Scans, and log shows 400 response for each
08-Dec-2021 13:38:03 : Start package process for: 6F541 08-Dec-2021 13:38:03 : Start transport process. 08-Dec-2021 13:38:03 : Token is valid 08-Dec-2021 13:38:03 : Getting authentication to upload file to Media Server... 08-Dec-2021 13:38:04 : Authentication granted, uploading file to Media Server... 08-Dec-2021 13:38:04 : Uploading file... 08-Dec-2021 13:38:05 : Uploaded file successfully 08-Dec-2021 13:38:05 : File uploaded, instruction Cloud to execute file... 08-Dec-2021 13:38:06 : Cloud succesfully instructed to execute file 08-Dec-2021 13:38:06 : File successfully uploaded (4CA9E_00002_2021-11-01_1259_0800.zip) 08-Dec-2021 13:38:07 : Token is valid 08-Dec-2021 13:38:07 : Getting authentication to upload file to Media Server... 08-Dec-2021 13:38:07 : Authentication granted, uploading file to Media Server... 08-Dec-2021 13:38:07 : Uploading file... 08-Dec-2021 13:38:08 : Uploaded file successfully 08-Dec-2021 13:38:08 : File uploaded, instruction Cloud to execute file... 08-Dec-2021 13:38:16 : Failed to execute upload 08-Dec-2021 13:38:16 : Exceptions: 08-Dec-2021 13:38:16 : Response status code does not indicate success: 400 (Bad Request). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<>c__DisplayClass17_0.<<ExecuteUpload>b__0>d.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 305 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext() in /_/src/Polly/Retry/AsyncRetryEngine.cs:line 69 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Polly.AsyncPolicy`1.<ExecuteAsync>d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<ExecuteUpload>d__17.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 302 08-Dec-2021 13:38:16 : Could not upload file 6F541_00004_2021-12-08_1330_0124.zip to cloud 08-Dec-2021 13:38:16 : Exceptions: 08-Dec-2021 13:38:16 : Failed to execute upload at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<ExecuteUpload>d__17.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 339 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<UploadFile>d__13.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 171 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Insight.Discovery.Logic.FileTransfer.TransferFile() in C:\Source\discovery\Discovery\Logic\FileTransfer.cs:line 199 08-Dec-2021 13:38:16 : Response status code does not indicate success: 400 (Bad Request). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<>c__DisplayClass17_0.<<ExecuteUpload>b__0>d.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 305 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext() in /_/src/Polly/Retry/AsyncRetryEngine.cs:line 69 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Polly.AsyncPolicy`1.<ExecuteAsync>d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<ExecuteUpload>d__17.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 302 08-Dec-2021 13:38:17 : Token is valid 08-Dec-2021 13:38:17 : Getting authentication to upload file to Media Server... 08-Dec-2021 13:38:17 : Authentication granted, uploading file to Media Server... 08-Dec-2021 13:38:17 : Uploading file... 08-Dec-2021 13:38:18 : Uploaded file successfully 08-Dec-2021 13:38:18 : File uploaded, instruction Cloud to execute file... 08-Dec-2021 13:38:22 : Failed to execute upload 08-Dec-2021 13:38:22 : Exceptions: 08-Dec-2021 13:38:22 : Response status code does not indicate success: 400 (Bad Request). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<>c__DisplayClass17_0.<<ExecuteUpload>b__0>d.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 305 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext() in /_/src/Polly/Retry/AsyncRetryEngine.cs:line 69 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Polly.AsyncPolicy`1.<ExecuteAsync>d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<ExecuteUpload>d__17.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 302 08-Dec-2021 13:38:22 : Could not upload file A66E9_00003_2021-10-19_1906_0671.zip to cloud 08-Dec-2021 13:38:22 : Exceptions: 08-Dec-2021 13:38:22 : Failed to execute upload at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<ExecuteUpload>d__17.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 339 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<UploadFile>d__13.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 171 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Insight.Discovery.Logic.FileTransfer.TransferFile() in C:\Source\discovery\Discovery\Logic\FileTransfer.cs:line 199 08-Dec-2021 13:38:22 : Response status code does not indicate success: 400 (Bad Request). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<>c__DisplayClass17_0.<<ExecuteUpload>b__0>d.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 305 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext() in /_/src/Polly/Retry/AsyncRetryEngine.cs:line 69 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Polly.AsyncPolicy`1.<ExecuteAsync>d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Insight.Discovery.Tools.Cloud.Services.InsightHttpService.<ExecuteUpload>d__17.MoveNext() in C:\Source\discovery\DiscoTools\Cloud\Services\InsightHttpService.cs:line 302 08-Dec-2021 13:38:22 : Finished transport process.
Workaround
Make sure you export one Scan Setting at a time - either by adjusting the Scanning times in Discovery, or, manually uploading the files / executing discovery.exe -z to export one file at a time
