Assets CVE Import process takes longer or hangs

XMLWordPrintable

    • 4
    • Severity 3 - Minor
    • 41

      Issue Summary

      Assets CVE Import hangs for hours without completion, causing high CPU usage. We observe the NVD API failure in the logs. 

      This bug is a placeholder for the CVE Import issues currently faced. The idea is to revisit it once the NVD API stabilizes.

      This is reproducible on Data Center: yes

      Steps to Reproduce

      1. Setup a CVE import process in Assets by specifying the publish start date and end date with the difference between dates greater than 100
      2. Create the structure and mapping and start the import process.

      Expected Results

      The import process completes within a short time less than an hour.

      Actual Results

      The import process continues to run for hours and sometimes never completes. All the while we see increased CPU usage.
      The below exception is thrown in the atlassian_jira.log file:

      2024-07-03 21:27:55,214+0530 insight-InsightImportThreadGroup-worker-thread-4 WARN admin     [i.r.j.p.c.client.interceptor.RetryInterceptor] Response <html><body><h1>503 Service Unavailable</h1>
    No server is available to handle this request.
    </body></html>

      This is primarily because there are stability issues with NVD API. The below error is observed even when connecting via postman.

       https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=0&pubStartDate=2020-01-01T00%3A00%3A00Z%5BUTC%5D&pubEndDate=2020-04-10T00%3A00%3A00Z%5BUTC%5D

<HTML>
<body>
<h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body>
</html>

       
      NVD website https://www.nist.gov/itl/nvd

       

      Workaround

      Try to use last modified days field instead of publish start and end dates. 

      Basically, if you put any number in the last modified field in the configuration, the date range settings will be ignored, and CVE import will poll only the CVEs that have been modified (or created) within the last X days.

      This will greatly limit the number of calls made to NVD and the CPU resources needed to load and assess objects already in the schema, which is up to 100% of every import currently performing.

       We suggest putting in there a number of +2 days since the last successful import and then keeping it synchronizing daily with a small number of days, e.g., 5. This should help workaround all the issues you are seeing and survive times of NVD instability like this one.

      Workaround to fetch inital data.

      This is to create initial CVE data as there is no offline mode now in newer versions.

      In this approach, the customer has to do multiple runs by changing the date range.
      Set the start date to 1988-10-01, and end date to 1995-10-01 . Add your API key in the API Key field.

      1. Leave the Last modified (in days) field empty.
      2. Run the import.

      Like wise, repeat the same process by increasing the date range by 5 years until 2005. From 2005, increase the date range by 2 years in each run, or if it is still slow, increase only by one year. Once all the data is fetched, then you can change the setup to fetch data from only the last X modified days, as specified in the first workaround of this bug.

        1. Screenshot 2024-07-16 at 2.56.43 PM.png
          Screenshot 2024-07-16 at 2.56.43 PM.png
          442 kB

              Assignee:
              Unassigned
              Reporter:
              Sireesha
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: