Uploaded image for project: 'Crowd Data Center'
  1. Crowd Data Center
  2. CWD-2809

Use a single query for cross-directory searches when all directories are local

    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      When querying across directories, Crowd currently queries them individually and then aggregates in-memory.

      When all directories are stored locally, either as local directories or cached remote, aggregating across directories can happen directly in the database for a performance increase.

            [CWD-2809] Use a single query for cross-directory searches when all directories are local

            Katherine Yabut made changes -
            Workflow Original: JAC Suggestion Workflow [ 3388600 ] New: JAC Suggestion Workflow 3 [ 3630700 ]
            Monique Khairuliana (Inactive) made changes -
            Workflow Original: Simplified Crowd Development Workflow v2 [ 1392008 ] New: JAC Suggestion Workflow [ 3388600 ]
            Issue Type Original: Improvement [ 4 ] New: Suggestion [ 10000 ]
            Status Original: Open [ 1 ] New: Gathering Interest [ 11772 ]
            Owen made changes -
            Workflow Original: Crowd Development Workflow v2 [ 379134 ] New: Simplified Crowd Development Workflow v2 [ 1392008 ]
            joe made changes -
            Link New: This issue is related to CWD-3225 [ CWD-3225 ]

            Aggregation in this case means two different things: sorting and removing shadowed users.
            Both of these have problems if you try to do them directly in the DB.
            Removing shadowed users I believe is too complex for direct DB manipulation and sorting could produce different results to today based on the collation.

            Furthermore some DAO implementations (JIRA) include in memory caches, so directly calling the DB would lead to a massive decrease in performance.

            Instead there are other inefficiencies that we could explore:

            • ApplicationServiceGeneric lowercases the usernames for duplicate detection
              We already store the lowercase name in the DB but we don't expose it through the User object
            • ApplicationServiceGeneric sorts Users even when there are is no "maxResults" in the query
              Sorting is only needed when paging.
              An ALL_RESULTS query is going to be big and is most in need of optimisation.
            • ApplicationServiceGeneric sorts all Users
              We only need to sort up to the end of the current page.
              Users that don't fall into that subset can be discarded without full sorting

            Mark Lassau (Inactive) added a comment - Aggregation in this case means two different things: sorting and removing shadowed users. Both of these have problems if you try to do them directly in the DB. Removing shadowed users I believe is too complex for direct DB manipulation and sorting could produce different results to today based on the collation. Furthermore some DAO implementations (JIRA) include in memory caches, so directly calling the DB would lead to a massive decrease in performance. Instead there are other inefficiencies that we could explore: ApplicationServiceGeneric lowercases the usernames for duplicate detection We already store the lowercase name in the DB but we don't expose it through the User object ApplicationServiceGeneric sorts Users even when there are is no "maxResults" in the query Sorting is only needed when paging. An ALL_RESULTS query is going to be big and is most in need of optimisation. ApplicationServiceGeneric sorts all Users We only need to sort up to the end of the current page. Users that don't fall into that subset can be discarded without full sorting
            joe made changes -
            Link New: This issue relates to CWD-2804 [ CWD-2804 ]
            Justin Koke made changes -
            Labels New: backlog
            joe created issue -

              Unassigned Unassigned
              jwalton joe
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: