Issue Summary

If permissions are changed to more lenient than 600 on one of the public key files (control-plane.pem or signing-key.pem) on a remote mesh node, GRPC operations from/to this mesh node start failing on restart. Even reverting the file permissions to original doesn't bring the mesh node back to normal state and the node has to be deleted and re-added to Bitbucket to bring it back to normalcy.

Steps to Reproduce

1. Setup 3 mesh nodes with the replication factor of 3 and add them to Bitbucket.
2. Create one or more repositories on the remote mesh nodes.
3. Change the file permissions to 640 on <MESH_HOME>/config/signing-key.pem on one of the mesh nodes.

• If you change permission on <MESH_HOME>/config/control-plane.pem instead, same issues will be observed although the exception messages will be different.

4. Restart the mesh node.
5. Try making some changes in one of the files in a repository through UI. Alternatively, push the changes using git push.

Expected Results

Changes are done successfully and synced on all the mesh nodes without any errors.

Actual Results

Errors encountered on all the mesh nodes and the file Edit fails on the UI.

The below warning message is shown during startup as expected -

2024-10-29 03:52:34,702 WARN  [main] - c.a.bitbucket.mesh.util.KeyUtils /home/ubuntu/mesh_home/config/signing-key.pem will not be read; its permissions are too loose (rw-r--r--)
2024-10-29 03:52:34,927 INFO  [main] - c.a.b.m.a.DefaultAuthenticationManager Generated signing key (Fingerprint: 1A6xIIeVKh/s+bnJ8U9Ek19XxM79K7UwUNIHLdLG6No)

The below exception is thrown in the Mesh application logs -
On the node where file permissions were changed -

2024-10-29 04:02:16,833 WARN  [grpc-client:thread-1] admin 7LS3HN8Qx242x13x2 @1MC9AIWx242x2922x0 10.229.140.128 "CommitService/EditFile" (>3 <1) c.a.b.m.t.LocalTransactionLeader [n/11/tx/74a751830000000000000002] Removed Node1@1 from the write lock because it failed to prepare
io.grpc.StatusRuntimeException: UNAUTHENTICATED: The [mesh-token] is invalid
	at io.grpc.Status.asRuntimeException(Status.java:537)

On other nodes -

2024-10-29 04:02:16,826 WARN  [grpc-server:thread-38894] 44BKQ23Mx242x15245x2 c.a.b.mesh.auth.JwtAuthenticator Rejecting token; signature verification failed due to missing public key
com.atlassian.bitbucket.mesh.auth.MissingNodeKeyException: No signing key could be resolved for 11 with fingerprint 1A6xIIeVKh/s+bnJ8U9Ek19XxM79K7UwUNIHLdLG6No
	at com.atlassian.bitbucket.mesh.auth.JwtAuthenticator$RegistrySigningKeyResolver.resolveSigningKey(JwtAuthenticator.java:202)
	at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:376)
	at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:550)
	at io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJws(DefaultJwtParser.java:610)

Workaround

For control-plane.pem
Stop the mesh node, change the file permission to 600 and restart the mesh node.

For signing-key.pem

1. Force delete the mesh node using this REST API with ?force=true
2. Add the node back using UI or using this REST API