Mesh: an object doesn't get distributed to other Mesh nodes

I’m Dmitry Pavlenko from TMate Software – a vendor developing SVN Mirror app for Bitbucket Server/Data Center. After migrating to Mesh (Bitbucket 8) our app stops working because some objects don’t get replicated to all Mesh nodes.

We’re heavily using

```
git hash-object -t <type> --stdin -w
```

commands (where <type> is “blob”, “tree”, or “commit”) to create objects in the Git repository while translating SVN to Git. Every time the command works fine and returns SHA-1 hash of the object just created. But approximately once in 8 runs the command works fine but creates the object only on one Mesh node and doesn’t replicate it to others (we tested a setup with 3 nodes). It doesn’t replicate the object neither immediately nor after waiting for several days, so it’s not a question of time. Though the most of the time (7 times of 8) everying is fine, the objects are replicated automatically.

This results into severe problems in our app because after running “git hash-object“, our app relies on the fact that the object (which SHA-1 is returned by that command) is already in the Git repository storage and at every moment can be accessed by “git cat-file“ command. But when the problem happens, it is not so, and it’s a question of luck whether at the moment of querying the object the query gets redirected to this or that Mesh node.

My question is whether this behaviour is expected?

If yes, what’s the recommended work-around for that? E.g. is there a Mesh API call that would enforce the object(s) replicaion?

If no, how can I help you to fix the problem (e.g. in Bitbucket)? It is eaily reproducible in our test setup.

Just to illustrate the problem on a certain example, I’m attaching a screenshot that I see when visiting the following [artificially constructed] URL:

https://MY_HOST/projects/ST/repos/sqljet2/commits/b37eb3015ac6d62d98b9ae2f52739cc07bf3d7b

The SHA-1 in the URL:
```
b37eb3015ac6d62d98b9ae2f52739cc07bf3d7b
```
is the commit that doesn’t get replicated properly (i.e. it is present on one Mesh node and is absent in another two). As you can see on the background, the commit message and its changes are loaded correctly. But then “git branch --color …“ command fails for this commit. This can be explained by the fact that

• the first query for the commit object gets redirected to one Mesh node (with the object – and its info is loaded correctly) while
• the second query gets redirected to another Mesh node where the commit object is absent.

When I refresh the page, I just get 404 error, because now the first query gets redirected to the second Mesh node (without the commit object). If I refresh this 404 page, again I get the page as on the screen shot attached.

Originally I reported the problem there: https://ecosystem.atlassian.net/servicedesk/customer/portal/9/AMKTHELP-44881

but then I've realised that it's a wrong place to report this bug.