Details
-
Suggestion
-
Resolution: Fixed
-
None
-
None
Description
Issue Summary
This is reproducible on Data Center: yes
After the check for errors phase, we proceed to the "Review your migration" phase, and this is where the problem happens. The estimated time keeps forever spinning, if I leave and let CCMA working, it will show a connection error eventually.
Environment information
Hosting (Server/DC/Cloud) server
Version 7.7.4
- of Projects / Spaces 530
- of Issues / Pages 20,552
- of Users 2,516
- of Groups 1,390
Attachments Size (in GB) 48
Operating System Linux CentOS
Database PostgreSQL 10.22
User Management Confluence internal
Hosting Location (Datacenter, AWS, etc) AWS
Integrations Jira Cloud
AWS EC2 c5.2xlarge (CPU 8, Memory 16GB)
CATALINA_OPTS= -Xms8192m -Xmx8192m
Steps to Reproduce
- Start a new CCMA migration with CCMA version 3.3.6
- Add some spaces
- Proceed to Review your migration
Expected Results
The migration plan should be created and the results of pre-checks should show in the page.
Actual Results
The estimated time keeps forever spinning, if I leave and let CCMA working, it will show a connection error eventually.
The rest of the instance gets unresponsive while CCMA is in that phase. For example, if I open another tab and try to navigate to any Confluence page, it will not work and will forever spin. If I close that CCMA tab, Confluence will get back to life at the same moment.
Checking the server resources allocation, we can see that there's a huge spike in the CPU, from 500% to 800%. Screenshots attached.
Checking the catalina logs, I can see several of these
17-Aug-2022 00:25:36.452 WARNING [Catalina-utility-4] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-23 url: /rest/migration/latest/stats/usersGroups; user: gmuller] (id=[799]) has been active for [67,496] milliseconds (since [8/17/22 12:24 AM]) to serve the same request for [http://34.211.15.211:8090/rest/migration/latest/stats/usersGroups] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [18] thread(s) in total that are monitored by this Valve and may be stuck. java.lang.Throwable at org.hibernate.event.internal.AbstractVisitor.processValue(AbstractVisitor.java:106)
The below exception is thrown in the confluence-atlassian.log file:
2022-08-09 16:33:19,921 ERROR [http-nio-8079-exec-4639] [[Standalone].[localhost].[/wiki].[servlet-module-container-servlet]] log Servlet.service() for servlet [servlet-module-container-servlet] threw exception -- referer: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=119683146 | url: /wiki/download/attachments/119683146/clg1.jpg | traceId: fbe931307b7bef1e | userName: dacoleman@mozilla.com java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe at com.atlassian.confluence.image.effects.ImageFilterServlet.uncheckAndIgnoreClientAbortExceptions(ImageFilterServlet.java:471) at com.atlassian.confluence.image.effects.ImageFilterServlet.lambda$doGet$0(ImageFilterServlet.java:150) at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21) at com.atlassian.sal.spring.component.SpringHostContextAccessor.lambda$doInTransaction$0(SpringHostContextAccessor.java:70) at ... com.atlassian.confluence.image.effects.ImageFilterServlet.lambda$doGet$0(ImageFilterServlet.java:148) ... 510 more Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:138) at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101) at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1252) at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:764) at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:584) at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:528) at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538) at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84) at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190) at org.apache.coyote.Response.doWrite(Response.java:601) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339) ... 525 more
Workaround
Upgrade CCMA to latest marketplace version and retry the migration