Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-13477

Repositories stop syncing on mirror due to an error

XMLWordPrintable

      Issue Summary

      An error during the sync of one repository on a mirror halts the sync of all the repositories for about 4 hours.

      This is reproducible on Data Center: yes

      Steps to Reproduce

      1. Setup Bitbucket and a single mirror
      2. Create one or more projects with few repositories.
      3. Add the projects created above on the mirror and wait for them to be synced
      4. Go to <MIRROR_BITBUCKET_HOME>/shared/data/repositories/<repo_id>/objects directory for one of the repositories and delete some directories while leaving the others. (This step is performed just to simulate the real world scenario where a repository can end up in a state where some object file is empty or it has a broken ref.)
      5. Push a change (using primary URL) in the repository used in above step.
      6. Start pushing changes in one or more of other repositories for more than a minute. (You can use a script that can keep creating branches in a loop and push them)
      7. Try to clone one of the repositories (not the one whose objects were deleted on disk) where changes where pushed in previous step.
      8. Check the contents of the cloned repository.

      Expected Results

      The cloned repository contains all the content that was pushed.

      Actual Results

      The cloned repository contains stale content as the repository is not synced on mirror. The repository can remain stale for up to 4 hours on the mirror.

      The below exception is thrown:

      2022-08-26 16:21:37,818 ERROR [farm-operation-3] @12RE2XVx969x255x0 c.a.b.i.m.m.f.t.ReplyingTopicListener [c8cc2baec3f35317b22ebc28725ab896] Failed to publish response to replyTopic:refChangeChunk_84932124-b53b-4d30-bdee-46f48f13f58a_reply  retries exhausted, giving up
      com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.atlassian.hazelcast.serialization.OsgiSafe'
      	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82)
      	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:157)
      	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133)
      	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118)
      	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:106)
      	at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:319)
      	at com.hazelcast.topic.impl.TopicService.publishMessage(TopicService.java:168)
      	at com.hazelcast.topic.impl.TopicProxySupport.publishInternal(TopicProxySupport.java:97)
      	at com.hazelcast.topic.impl.TopicProxy.publish(TopicProxy.java:38)
      	at com.atlassian.stash.internal.topic.HazelcastTopicService$DefaultTopic.publish(HazelcastTopicService.java:258)
      	at com.atlassian.bitbucket.internal.mirroring.mirror.mdc.StateTransferringTopic.publish(StateTransferringTopic.java:30)
      	at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener.publishResponse(ReplyingTopicListener.java:103)
      	at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener.access$100(ReplyingTopicListener.java:28)
      	at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener$1.onFailure(ReplyingTopicListener.java:71)
      	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
      	at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.lang.Thread.run(Thread.java:748)
      	... 1 frame trimmed
      Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.atlassian.bitbucket.internal.mirroring.mirror.mdc.StatefulObject'
      	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82)
      	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:254)
      	at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:376)
      	at com.atlassian.hazelcast.serialization.OsgiSafeStreamSerializer.write(OsgiSafeStreamSerializer.java:122)
      	at com.atlassian.hazelcast.serialization.OsgiSafeStreamSerializer.write(OsgiSafeStreamSerializer.java:63)
      	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43)
      	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:154)
      	... 17 common frames omitted
      Caused by: java.io.NotSerializableException: io.grpc.Status
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
      	at java.lang.Throwable.writeObject(Throwable.java:1024)
      	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
      	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:116)
      	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:109)
      	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43)
      	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:252)
      	... 22 common frames omitted
      

      Workaround

      Move the repository which ends up in an error state (e.g. due to broken ref) to another project which is not added in mirror sync list. If all the projects are synced on the mirror then the only work-around is to set mesh.enabled=false in the bitbucket.properties file on the mirror and restart it.

              mgoyal2@atlassian.com Manish
              mgoyal2@atlassian.com Manish
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: