Loading...

Details

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 8.0.4, 8.1.4, 8.2.3, 8.3.2, 8.4.1
Affects Version/s: 8.0.0, 8.1.0, 8.2.0, 8.3.0, 8.1.3
Component/s: Smart Mirroring
Labels:
None

Symptom Severity:
Severity 1 - Critical
Bug Fix Policy:
View Atlassian Server bug fix policy

Description

Issue Summary

An error during the sync of one repository on a mirror halts the sync of all the repositories for about 4 hours.

This is reproducible on Data Center: yes

Steps to Reproduce

Setup Bitbucket and a single mirror
Create one or more projects with few repositories.
Add the projects created above on the mirror and wait for them to be synced
Go to <MIRROR_BITBUCKET_HOME>/shared/data/repositories/<repo_id>/objects directory for one of the repositories and delete some directories while leaving the others. (This step is performed just to simulate the real world scenario where a repository can end up in a state where some object file is empty or it has a broken ref.)
Push a change (using primary URL) in the repository used in above step.
Start pushing changes in one or more of other repositories for more than a minute. (You can use a script that can keep creating branches in a loop and push them)
Try to clone one of the repositories (not the one whose objects were deleted on disk) where changes where pushed in previous step.
Check the contents of the cloned repository.

Expected Results

The cloned repository contains all the content that was pushed.

Actual Results

The cloned repository contains stale content as the repository is not synced on mirror. The repository can remain stale for up to 4 hours on the mirror.

The below exception is thrown:

2022-08-26 16:21:37,818 ERROR [farm-operation-3] @12RE2XVx969x255x0 c.a.b.i.m.m.f.t.ReplyingTopicListener [c8cc2baec3f35317b22ebc28725ab896] Failed to publish response to replyTopic:refChangeChunk_84932124-b53b-4d30-bdee-46f48f13f58a_reply  retries exhausted, giving up
com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.atlassian.hazelcast.serialization.OsgiSafe'
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:157)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:106)
	at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:319)
	at com.hazelcast.topic.impl.TopicService.publishMessage(TopicService.java:168)
	at com.hazelcast.topic.impl.TopicProxySupport.publishInternal(TopicProxySupport.java:97)
	at com.hazelcast.topic.impl.TopicProxy.publish(TopicProxy.java:38)
	at com.atlassian.stash.internal.topic.HazelcastTopicService$DefaultTopic.publish(HazelcastTopicService.java:258)
	at com.atlassian.bitbucket.internal.mirroring.mirror.mdc.StateTransferringTopic.publish(StateTransferringTopic.java:30)
	at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener.publishResponse(ReplyingTopicListener.java:103)
	at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener.access$100(ReplyingTopicListener.java:28)
	at com.atlassian.bitbucket.internal.mirroring.mirror.farm.topic.ReplyingTopicListener$1.onFailure(ReplyingTopicListener.java:71)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
	at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.lang.Thread.run(Thread.java:748)
	... 1 frame trimmed
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.atlassian.bitbucket.internal.mirroring.mirror.mdc.StatefulObject'
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:254)
	at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:376)
	at com.atlassian.hazelcast.serialization.OsgiSafeStreamSerializer.write(OsgiSafeStreamSerializer.java:122)
	at com.atlassian.hazelcast.serialization.OsgiSafeStreamSerializer.write(OsgiSafeStreamSerializer.java:63)
	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:154)
	... 17 common frames omitted
Caused by: java.io.NotSerializableException: io.grpc.Status
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
	at java.lang.Throwable.writeObject(Throwable.java:1024)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:116)
	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:109)
	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:252)
	... 22 common frames omitted

Workaround

Move the repository which ends up in an error state (e.g. due to broken ref) to another project which is not added in mirror sync list. If all the projects are synced on the mirror then the only work-around is to set mesh.enabled=false in the bitbucket.properties file on the mirror and restart it.

Attachments

Issue Links

mentioned in: Page Loading...

Repositories stop syncing on mirror due to an error