-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Highest
-
Affects Version/s: 8.0.0, 8.14.0, 8.9.6, 8.14.2, 8.17.0
-
Component/s: Git Hosting
-
13
-
Severity 2 - Major
-
349
-
CtB - Improve Existing
Issue Summary
The mesh sidecar is shut down by “mesh-sidecar-monitor-thread” after a number of SQL server connectivity errors and never started again.
This is reproducible on Data Center: (yes)
Steps to Reproduce
- # Launch Bitbucket 8.9.6 and connect it to external database
- Shut down PostgreSQL server.
- Wait around ~50 minutes, and "mesh-sidecar-monitor" will try to restart mesh sidecar, but will actually only shut it down:
c.a.s.i.s.g.m.DefaultSidecarManager gRPC ping response hasn't been received for 5 periods (100041ms ago), attempting restart
Expected Results
Mesh sidecar to be up and running although there were SQL server errors.
Actual Results
Mesh sidecar is shut down and stays off until Bitbucket is restarted.
mesh-sidecar-monitor messages in the atlassian-bitbucket.log file
2024-01-08 17:09:41,447 WARN [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08001 2024-01-08 17:09:41,447 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms. 2024-01-08 17:09:41,447 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper The connection attempt failed. 2024-01-08 17:09:41,454 INFO [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Failed to send gRPC ping 2024-01-08 17:09:56,460 WARN [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08001 2024-01-08 17:09:56,461 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms. 2024-01-08 17:09:56,461 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper The connection attempt failed. 2024-01-08 17:09:56,463 INFO [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Failed to send gRPC ping 2024-01-08 17:10:11,465 WARN [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08001 2024-01-08 17:10:11,465 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms. 2024-01-08 17:10:11,465 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper The connection attempt failed. 2024-01-08 17:10:11,467 INFO [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Failed to send gRPC ping 2024-01-08 17:10:26,471 WARN [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08001 2024-01-08 17:10:26,471 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15001ms. 2024-01-08 17:10:26,472 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper The connection attempt failed. 2024-01-08 17:10:26,475 INFO [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Failed to send gRPC ping 2024-01-08 17:10:41,484 WARN [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08001 2024-01-08 17:10:41,484 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15001ms. 2024-01-08 17:10:41,484 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper The connection attempt failed. 2024-01-08 17:10:41,493 INFO [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Failed to send gRPC ping 2024-01-08 17:10:56,497 WARN [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08001 2024-01-08 17:10:56,497 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms. 2024-01-08 17:10:56,497 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper The connection attempt failed. 2024-01-08 17:10:56,499 INFO [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Failed to send gRPC ping 2024-01-08 17:10:56,501 WARN [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager gRPC ping response hasn't been received for 5 periods (100041ms ago), attempting restart 2024-01-08 17:10:57,397 INFO [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Sidecar has stopped (Exit code: 0) 2024-01-08 17:11:12,400 WARN [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: 08001 2024-01-08 17:11:12,401 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms. 2024-01-08 17:11:12,401 ERROR [mesh-sidecar-monitor:thread-2] o.h.e.jdbc.spi.SqlExceptionHelper The connection attempt failed. 2024-01-08 17:11:12,404 ERROR [mesh-sidecar-monitor:thread-2] c.a.s.i.s.g.m.DefaultSidecarManager Unhandled exception caught
No mesh sidecar running after the failed restart:
root 27349 0.0 0.2 1459116 19728 ? Sl 17:09 0:02 /usr/bin/containerd-shim-runc-v2 -namespace moby -id ce1867bf734fc4bb5808cea304a2d3c24317d987fd4e4bf011bd6e86b424ea9e -address /var/run/docker/containe root 27369 0.0 0.0 2204 776 ? Ss 17:09 0:00 \_ /usr/bin/tini -- /entrypoint.py --log=INFO 2003 27426 7.1 20.2 5415112 1648292 ? Sl 17:09 6:05 \_ /opt/java/openjdk/bin/java -classpath /opt/atlassian/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/var/atlassian/application
Workaround
Currently there is no known workaround for this behavior. Bitbucket must be restarted after mesh sidecar is shut down.