High Issue Summary
An error during the sync of one repository on a mirror halts the sync of all the repositories for about 4 hours.
This is reproducible on Data Center: yes
Steps to Reproduce
- Setup Bitbucket and a single mirror
- Create one or more projects with few repositories.
- Add the projects created above on the mirror and wait for them to be synced
- Go to <MIRROR_BITBUCKET_HOME>/shared/data/repositories/<repo_id>/objects directory for one of the repositories and delete some directories while leaving the others. (This step is performed just to simulate the real world scenario where a repository can end up in a state where some object file is empty or it has a broken ref.)
- Push a change (using primary URL) in the repository used in above step.
- Start pushing changes in one or more of other repositories for more than a minute. (You can use a script that can keep creating branches in a loop and push them)
- Try to clone one of the repositories (not the one whose objects were deleted on disk) where changes where pushed in previous step.
- Check the contents of the cloned repository.
Expected Results
The cloned repository contains all the content that was pushed.
Actual Results
The cloned repository contains stale content as the repository is not synced on mirror. The repository can remain stale for up to 4 hours on the mirror.
The below exception is thrown:
2022-08-26 16:21:37,818 ERROR [farm-operation-3] @12RE2XVx969x255x0 c.a.b.i.m.m.f.t.ReplyingTopicListener [c8cc2baec3f35317b22ebc28725ab896] Failed to publish response to replyTopic:refChangeChunk_84932124-b53b-4d30-bdee-46f48f13f58a_reply retries exhausted, giving up com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.atlassian.hazelcast.serialization.OsgiSafe' at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82) at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:157) at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133) at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118) at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:106) at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:319) at com.hazelcast.topic.impl.TopicService.publishMessage(TopicService.java:168) at com.hazelcast.topic.impl.TopicProxySupport.publishInternal(TopicProxySupport.java:97) at com.hazelcast.topic.impl.TopicProxy.publish(TopicProxy.java:38) at com.atlassian.stash.internal.topic.HazelcastTopicService$DefaultTopic.publish(HazelcastTopicService.java:258) at com.atlassian.bitbucket.internal.mirroring.mirror.mdc.StateTransferringTopic.publish(StateTransferringTopic.java:30) at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener.publishResponse(ReplyingTopicListener.java:103) at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener.access$100(ReplyingTopicListener.java:28) at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener$1.onFailure(ReplyingTopicListener.java:71) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125) at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.lang.Thread.run(Thread.java:748) ... 1 frame trimmed Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.atlassian.bitbucket.internal.mirroring.mirror.mdc.StatefulObject' at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82) at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:254) at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:376) at com.atlassian.hazelcast.serialization.OsgiSafeStreamSerializer.write(OsgiSafeStreamSerializer.java:122) at com.atlassian.hazelcast.serialization.OsgiSafeStreamSerializer.write(OsgiSafeStreamSerializer.java:63) at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43) at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:154) ... 17 common frames omitted Caused by: java.io.NotSerializableException: io.grpc.Status at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441) at java.lang.Throwable.writeObject(Throwable.java:1024) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:116) at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:109) at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43) at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:252) ... 22 common frames omitted
Workaround
Move the repository which ends up in an error state (e.g. due to broken ref) to another project which is not added in mirror sync list. If all the projects are synced on the mirror then the only work-around is to set mesh.enabled=false in the bitbucket.properties file on the mirror and restart it.