Hello,
during a loadtest we are running on our new production system (still not productive to end users), we see search response times are increasing to very high figures, up to 5 minutes.

We have very low CPU utilization on all app nodes and DB node.
We see all other requests are responding fast (few seconds) but search is really slow.

I took several thread dumps and all of them shows the following stack trace:
The call stack is hanging on the following for up to a few minutes:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service.poll(Service.CDB:31)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.requestLock(ReplicatedCache.CDB:42)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.lockResource(ReplicatedCache.CDB:89)
com.tangosol.coherence.component.util.CacheHandler.lock(CacheHandler.CDB:3)
com.tangosol.util.InvocableMapHelper.invokeLocked(InvocableMapHelper.java:58)
com.tangosol.coherence.component.util.CacheHandler.invoke(CacheHandler.CDB:3)
com.tangosol.coherence.component.util.SafeNamedCache.invoke(SafeNamedCache.CDB:1)
com.atlassian.confluence.cache.tangosol.hibernate.CoherenceInvokableHibernateCache.invoke(CoherenceInvokableHibernateCache.java:127)
com.atlassian.hibernate.extras.tangosol.CoherenceCacheStrategy.put(CoherenceCacheStrategy.java:175)
com.atlassian.confluence.cache.hibernate.ConfluenceCacheStrategy.put(ConfluenceCacheStrategy.java:68)
net.sf.hibernate.impl.SessionImpl.initializeEntity(SessionImpl.java:2243)
net.sf.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:319)
net.sf.hibernate.loader.Loader.doQuery(Loader.java:309)
net.sf.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:138)
net.sf.hibernate.loader.Loader.doList(Loader.java:1063)
net.sf.hibernate.loader.Loader.list(Loader.java:1048)
net.sf.hibernate.hql.QueryTranslator.list(QueryTranslator.java:854)
net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1569)
net.sf.hibernate.impl.QueryImpl.list(QueryImpl.java:49)
com.atlassian.confluence.spaces.persistence.dao.hibernate.HibernateSpaceDao$6.doInHibernate(HibernateSpaceDao.java:436)
org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:337)
com.atlassian.confluence.spaces.persistence.dao.hibernate.HibernateSpaceDao.getSpaces(HibernateSpaceDao.java:425)
com.atlassian.confluence.spaces.DefaultSpaceManager$1.getElements(DefaultSpaceManager.java:628)
com.atlassian.confluence.core.DefaultListBuilder.getPage(DefaultListBuilder.java:47)
com.atlassian.confluence.spaces.DefaultSpaceManager.getAllSpaces(DefaultSpaceManager.java:639)
sun.reflect.GeneratedMethodAccessor1487.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:16)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
com.sun.proxy.$Proxy57.getAllSpaces(Unknown Source)
com.atlassian.confluence.search.SpacePickerHelper.getAvailableGlobalSpaces(SpacePickerHelper.java:75)
com.atlassian.confluence.search.actions.SearchSiteAction.getAvailableGlobalSpaces(SearchSiteAction.java:448)

Any known issue with similar stack trace?

This is critical because the go-live date is Sunday and we will not be able to go live with such a performance.

Best,