Medium Issue Summary
gRPC messages between the Bitbucket webapp and Mesh sidecar exceeding the max size (5242880) raise RESOURCE_EXHAUSTED exceptions.
This is caused by the Commit Indexer sending too large includes and excludes for the GetCommits RPC.
This is reproducible on Data Center: yes
Expected Results
gRPC messages exceeding the default max size (5242880) are properly handled.
Actual Results
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED exceptions are raised.
The following warning is logged in the atlassian-mesh.log file
2024-05-29 09:15:15,971 WARN [grpc-nio-worker-ELG-3-1] - io.grpc.netty.NettyServerStream Exception processing message io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 5242880: 8492792 at io.grpc.Status.asRuntimeException(Status.java:526) at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:391) at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:271) at io.grpc.internal.MessageDeframer.deframe(MessageDeframer.java:177) at io.grpc.internal.AbstractStream$TransportState.deframe(AbstractStream.java:210) at io.grpc.internal.AbstractServerStream$TransportState.inboundDataReceived(AbstractServerStream.java:255) at io.grpc.netty.NettyServerStream$TransportState.inboundDataReceived(NettyServerStream.java:226) at io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:508) at io.grpc.netty.NettyServerHandler.access$900(NettyServerHandler.java:106) at io.grpc.netty.NettyServerHandler$FrameListener.onDataRead(NettyServerHandler.java:840) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307) at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829)
Workaround
- Edit $BITBUCKET_HOME/shared/bitbucket.properties
- Add the property:
plugin.bitbucket-git.mesh.grpc.max-message-size=<higher value>
The value should be higher than the value reported in the exception, e.g. 8492792 in the sample above
- Call the following endpoint to update the property value on the sidecar
curl -k -u <BITBUCKET_ADMIN_USER>:<PASSWORD> -X POST -H "Content-Type: application/json" <BITBUCKET_BASE_URL>/rest/ui/admin/git/mesh/config/refresh - Restart Bitbucket
Note: keep an eye on higher memory usage due to the bump
