Problem

According to the KB article for the Data Pipelines feature:

The export will include all data, including PII (Personally Identifiable Information) and restricted content. This is to provide you with as much data as possible, so you can filter and transform to generate the insights you’re after.

If you need to filter out data based on security and confidentiality, this must be done after the data is exported.

Exported files are saved in your shared home directory, so you’ll also want to check this is secured appropriately. 

This has the potential of being disallowed in highly-regulated environments where sensitive information cannot be exfiltrated.

Suggested Solution

The way we're currently envisioning this is by applying user-level permissions to the pipeline, and all information that is in-scope to that user's global, project, and issue permission levels passes through the filter.

Why This is Important

There may be instances where Jira resides in a highly-regulated environment, in which sensitive data cannot be exfiltrated to be post-processed off-site. In situations such as this, it could be beneficial to have a method of pre-filtering data before exporting via Data Pipelines.

Alternatives to Data Pipelines to populate a data lake for BI consumption (e.g. partial DB replication) are overly complex and/or require significant custom development.

Workaround

None if highly-regulated environment prohibits exfiltration of some data