-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
9.4.9
-
2
-
Severity 3 - Minor
-
42
-
Issue Summary
When Remote Mesh Nodes start up, they sometimes encounter the error "Rejecting token; signature verification failed due to missing public key com.atlassian.bitbucket.mesh.auth.MissingNodeKeyException: No signing key could be resolved for <NODE_ID> with fingerprint <fingerprint>."
This suggests that the public key of the other Mesh Nodes is missing, and the authentication to them is failing.
Steps to Reproduce
- Restart one of the Mesh Nodes while other Mesh Nodes are online.
- During startup, the Mesh Node sometimes throws the MissingNodeKeyException.
Expected Results
The other Mesh Nodes are up and running, and the Mesh Node starting up should be able to communicate with the other Nodes without any issues.
Improve logging to not show the Stack Trace and with meaningful Log Entries.
Actual Results
2025-08-13 03:01:56,487 WARN [grpc-server:thread-1] AM79HEVRx181x2x1 c.a.b.mesh.auth.JwtAuthenticator Rejecting token; signature verification failed due to missing public key com.atlassian.bitbucket.mesh.auth.MissingNodeKeyException: No signing key could be resolved for 2 with fingerprint AqIpij4TgjTYqOgYDOfs04NuVxxUroFal5h/3nfu7Ek at com.atlassian.bitbucket.mesh.auth.JwtAuthenticator$RegistrySigningKeyResolver.resolveSigningKey(JwtAuthenticator.java:202) at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:376) at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:550) at io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJws(DefaultJwtParser.java:610) at io.jsonwebtoken.impl.ImmutableJwtParser.parseClaimsJws(ImmutableJwtParser.java:173) at com.atlassian.bitbucket.mesh.auth.JwtAuthenticator.authenticate(JwtAuthenticator.java:71) at com.atlassian.bitbucket.mesh.auth.JwtAuthenticator$$FastClassBySpringCGLIB$$98da7f55.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at ----- at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)
Workaround
This issue is due to a temporary startup problem. Mesh nodes communicate with each other and authenticate messages using JWTs, which requires the public keys of the other mesh nodes. When a mesh node starts, it registers itself with the control plane. Subsequently, Bitbucket sends a "ManagementService/SetConfiguration" request containing the public keys of all Mesh nodes in the topology.
The problem occurs when a request from a peer Mesh node arrives before the "ManagementService/SetConfiguration" message has been fully processed. As a result, the authentication fails because the public keys are not yet available. This issue typically resolves itself within a few seconds once the configuration is completed and the necessary public keys are available.