Activity stream in the user profile times out easily on larger instances

Summary

Accessing a user profile, for example https://jira.atlassian.com/secure/ViewProfile.jspa, takes a long period of time to return results. If it consistently causes timeouts the Activity Stream may ban it as a provider, meaning streams may no longer return results for a period of time.

Environment

A larger instance for example 300,000 issues & 1000 projects.

Steps to Reproduce

Browse to a user's profile, for example https://jira.atlassian.com/secure/ViewProfile.jspa.

Expected Results

The results return quickly.

Actual Results

The results take > 10 seconds to return so a timeout is encountered. The below is encountered in atlassian-jira.log:

[atlassian.streams.internal.StreamsCompletionService] Registering failure for stream provider JIRA due to cancellation (timeout)

Timeouts

A timeout is triggered when the activity streams are not returned within the default timeout interval. This can be caused by any of the following:

• The JIRA instance is performing slowly when retrieving results (it may be under load or experiencing a performance problem).
• One of the providers in the stream is responding slowly.
• Something on the network layer is taking a long time.
• The index read speed may be slow (see our Test the Disk Speed KB)
• The provider may not be up and running.

Notes

It is possible for certain Activity Stream providers (Atlassian apps or plugin-specific apps accessible through AppLinks) to be banned from returning results if they consistently time out. This bug fix alleviates some impact of that as it will reduce timeouts on the user profiles within JIRA. If a provider is banned, there is not currently any warning of this - the fix to that is tracked in STRM-2264.

Additionally more information can be found about timeout behaviours in JIRA Activity Stream returns One or more activity sources were slow to respond and are not included.

Workaround

Prior to Activity Streams 5.4.x, it's possible to modify the query string using a front-end proxy such as Apache to rewrite it to be higher. This addresses the symptom rather than the cause (which is the change history results take a long time to return from the database). Add the following to the Apache configuration:

    RewriteEngine on
    RewriteCond %{REQUEST_URI} ^/plugins/servlet/streams
    RewriteCond %{QUERY_STRING} !^timeout.*$
    RewriteRule ^(.*)$ %{REQUEST_URI}?timeout=20000&%{QUERY_STRING} [R,L]

If the provider has been banned, and is no longer returning results, either waiting > 24 hours or restarting the instance will address that problem.