Problem
When performing a paged search from n to p using CrowdService.search(Query), Crowd internally:
- fetches all the results from 0 to p,
- reorders the results in memory,
- and skips the first n results to return a page seemingly from n to p.
The issue is that, when the API is used to fetch successive pages (ex: from 0 to p, then from p+1 to 2p and so on), the in-memory sort is performed on a larger set on the successive pages, from 0 to the actual limit (ex: 2p for the second paged search).
As a result, the same result(s) can have different position(s) in each successive page request. And, since Crowd silently discards the sorted results up to the start index of the requested page, those results can be silently dropped depending on their position.
Example
On a customer dataset, Stash performed a group membership query using paged searches, where it asked for all the users in a group groupA (direct and nested membership). Because the groupA has several nested groups with large numbers of direct users, the user foo would not be found within the first request, from 0 to 100. In the subsequent search (from 100 to 200), the user will be found in the position 115 (before the in-memory sort) and 81 (after the sort). As a result, when Crowd discarded the first 100 results, that user (and other ones) would be silently discarded. Hence, in the aggregated result, the user would not be present and the list of members of groupA returned to the caller would not include the user foo. For the same reason, other users would be duplicated in the aggregated result, which caused STASH-3843.
- causes
-
BSERV-3947 Searching nested group memberships fails for users with a large number of groups
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
- mentioned in
-
![[Extranet] Page](/images/icons/generic_link_16.png "[Extranet] Page")
Page
Failed to load
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
[CWD-3628] Paged searches with CrowdService.search give inconsistent results across pages
ApplicationServiceGeneric#searchNestedGroupRelationships is using the provided page size in the underlying query. It should request everything, then rely on constrainResults to select the appropriate page.
jwalton ResultsAggregator's AggregatorImpl implementation is the code path in Crowd 2.6.4 that creates the issue. In its implementation of the method constrainResults(), it sorts the results in memory and discards the results before the startIndex (which itself is provided by the query passed in CrowdService.search(Query), though ResultsAggregator.with.
CWD-2807 fixed a lot of these; in particular, ApplicationServiceGeneric#searchNestedGroupRelationships uses a ResultsAggregator. Is this an edge case, a missed fix or a different code path?
Note that the user would not be found because Crowd performs a BFS of the user memberships, in DirectoryManagerGeneric.findNestedUserMembersOfGroup().
To clarify for anyone who runs into this later, the issue description here is misleading, because you don't need nested groups at all to see this problem.
For example, to reproduce this, all you need to do is have an uncached ldap directory with 10 users, called user-1 to user-10 and an application configured with a login of myapp / myapp which is only mapped to that directory. Then rename user-10 to aaaa in the LDAP directory (may not be strictly necessary, but the goal is to get the ldap server to provide out of order results; don't forget to rename the attribute on the group as well!), and run
You'll see that some users show up twice, and others don't show up at all.
(you can see the same problem with less users, but 10 is a nice balance between easy-to-spot-when-it's-wrong and so-small-that-the-issue-is-rarely-seen)