Since JQL functions work as macros, whenever they return a big list JIRA needs to go through the whole list and validate the interim results before proceeding with the query.
Even though query is limited by other clauses, but JQL doesn't have smarts and calculates the context, even though a much more restricted context is provided by another clause.
Some improvements have been made to reduce the pain by batching and doing less work, but it doesn't scale beyond 100-200K issues.
We need to provide means to functions to override some of this behaviour.

Partial fix

First step to address this problem was done in JRASERVER-63665
JIRA prepares a search with a small limit set (1000) and check if totalHits is bigger. If it's bigger then we do a second search and we exactly know the limit now (totalHits from the previous search).

Additional logging

Additional logs can help to find a plugin which is doing such searches. To enable this:

• Set com.atlassian.jira.issue.search.providers.LuceneSearchProvider to DEBUG level to capture expensive JQLs:
  • It logs only queries which try to fetch more than 1000 documents.
  • Example of the logging:

    "Search for the query <> was performed and results count <> exceeds max allowed search limit 
    Large result sets could lead to OutOfMemoryError. If you're sure you need to do the query you can remove the limit by enabling feature."
    
    or 
    
    "Search for the query [<>] was performed and results count (<>) exceeds default limit"