All the data is found under ticket https://getsupport.atlassian.com/servicedesk/customer/portal/14/CSP-266691

Hi Maksym Fedoryshyn, we upgraded to Confluence version 7.3, to have this fix, then we managed to migrate all Gliffy diagrams to Gliffy on Friday. Today we have the following behaviour:

• when a page is not in Edit mode: it displays a Lucidchart diagram, and one, by clicking on the diagram is redirected to a Lucidchart diagram
• when a page is in Edit mode, it is a Gliffy diagra, and by by clicking on the diagram, a Gliffy edit mode diagram is opened

When we migrate the diagram again, the it is Lucidchart in the edit mode as well. But until when? When will it be changed back?

This is a serious issue, and it is affecting our Confluence database integrity. Please advise us what we could to to fix the issue.

Thank you.

Hi eva.dezsi, we got the same issue with 6.15.8, and it seems occur regularly, almost every Friday afternoon. There is nothing to do to fix it but restart service. Need your help, thanks!

Just wanted to share that this affects the Comala Publishing app. That allows you to publish from one space (draft) to another (target).

It was intermittent, but the workaround was to retry 20+ times or just delete and purge the page in the target space. As we saw it more, we finally opened a ticket with Comala and within 24 hours they reviewed logs and linked it to this bug. Wanted to share so that others searching with a similar issue can find this page.

Our route forward is to upgrade to 7.3.3. We had it in TEST and were evaluating before finding this out anyways, so it works for us.

Hi eva.dezsi, thank you for letting us know that this is still an issue for you.

Can you please provide more details on the issue with migration? This will help us to better understand the problem. Some useful items would be:

• logs with stack traces for errors during migration
• what confluence version is used?
• steps to reproduce (pls. include links to all plugins involved in the migration, please also mention their versions)
• dataset which can help to reproduce the issue (space/site exp. etc.)

Thank you,
-Maksym Fedoryshyn.

For us the fix did not solve the issue, Gliffy to Lucidchart migration is still unsuccessful.

A fix for this issue is available to Server and Data Center customers in Confluence 7.3.3
Upgrade now or check out the Release Notes to see what other issues are resolved.

Maksym Fedoryshyn it is great you managed to fix it. Could you please share potential release date? At least rough estimate. We are desperately waiting for this fix! Thank you.

Confirming on my side too, my automated page update script now works like a charm on my test server with the 7.4.0-m05 version.
Looking forward to viewing the official release!

Thanks for fixing it. We have already tested Scroll Versions with the new milestone and it seems to work  We are really happy about that! 

Hi All,

We have prepared a fix for this issue and we have made this fix available in the Confluence EAP release 7.4.0-m05. You can download this release from our EAP downloads page https://www.atlassian.com/software/confluence/download-eap.

We went through all cases reported for this issue (at least all we were able to reproduce), and fixed it by introducing a new recovery mechanism for unreconciled page updates. Recovery should happen transparently for all callers, so the API should behave as it used to before 7.0 (but now without danger of losing unpublished changes). Please give it a try and provide feedback.

On a side note please keep in mind, that in rare cases ExternalChangesException can still happen in the case where content update fails for some reason, so please don't forget to handle it appropriately. The fix also includes changes to solve the problem with visibility of ExternalChangesException class in the plugins and changes to the confusing exception message.

Lastly, if you need more fine access over the content reconciliation process, please consider using PageUpdateService. Its prepare() method can be used to ensure that content and draft are synchronized and are ready for updates. Methods of this service can be called within as well as outside of the active database transaction.

Thank you for your patience and we look forward to receiving your feedback on this fix.

Regards,

The Confluence Team

Hi All,
We have listened to your feedback and have been working on a fix. This solution aims to minimize changes necessary for apps running into this issue. Current work is wrapping up and we aim to ship it as part of an EAP in the coming weeks. More technical details will be available when releasing the fix. We will keep you updated on this bug report as we near this release and would love you to try out these changes to ensure it works for your use case.
Regards,
The Confluence Team

Can the guidance please be updated with something that works 100% of the time?

As I noted in https://jira.atlassian.com/browse/CONFSERVER-59196?focusedCommentId=2348467&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2348467, there are too many "should" statements and the retry time delay is not working reliably. This has been noted by others.

Given that the upcoming Confluence 7.4 will be a new enterprise release I think many larger instances will migrate directly from 6.6 / 6.13 to 7.4, hitting this bug.
Will it be fixed before the 7.4 release?

PLease fix this bug fast !

br mike

Hello 81f99ea8e422,

The workaround of retrying the does not always work in setting the stringval in contentproperties back from "restored".
In this case, as you're on Confluence 7.0.2, the suggested workaround would be to turn off collaborative editing, and turning it back on, which will clean up the table.

If the workaround above does not work for you on Confluence 7.0.2, do reach out to our support team using the link below, and we'll look into the issue further.

• Contact support

Cheers,
Sattesh

On 7.0.2 the suggested workaround of retrying after 100ms does not work, nor 200ms, nor any other retry interval and combination of retries. 

Hi,

found this affects Draw.io also in bulk imports.

Retrying in a separate transaction does not sound like a workaround - we will end up in data inconsistencies all the time.

For draw.io migration users, we've a route that is stable. Contact support@draw.io and we'll explain the process.

@atlassian, that doesn't remove our vote, the workaround is very expensive on engineer time, but I understand the issue is complex and you don't deem it high enough prority.

@Rupert Poon thanks for your analysis. The impact is much bigger and we are not the only vendor affected by this bug. That's why I do not understand why Atlassian is not responding here. We are losing trust from our customers and we have done everything possible to help Atlassian reproduce the bug. What is the current status?

Impact analysis for K15t Scroll Versions users: There are 1,488 live, paid for installations of Scroll Versions according to the Marketplace page. If we assume that they all want to use Confluence 7.x, and that the average installation has 50 users, that is 74,400 affected users. That's quite a lot of unhappy customers. Obviously, @Daria Huber at K15t can probably calculate the number of affected users from their own metrics.

Hello
we were just about to plan our migration from 6.3.1 which is outdated to 7.1.2, and this is clearly a show-stopper for us as well.
We have automated tasks that update documentation pages every day/week, and performing no-regression tests in my pre-prod instance, I just encountered this issue.
As mentioned by other people here, the workaround does not work (tried to view page, call several times the REST API with delay... => none worked). And anyway this is not what I'd expect as a "solution"...
This is a blocker for us too, and this means a no-go so far for our migration.
Regards

I agree, we need  to get an acknowledgement of the status within Atlassian. We are one of those affected customers - we are migrating from Gliffy to Lucidchart, and wre are now stuck between two solutions because the migration scenario doesn't work anymore. Can we get an estimated time when this will be fixed? Or Atlassian can please let us know what is needed to change the status from current status of 'Gathering Impact' to 'In Progress'? How many customers need to be affected?  Thank you

For REST API, the best option would be to re-try the operation. As mentioned above, the publish/update attempt will trigger the reconciliation process if the page is un-synchronized. A re-try of the same request should succeed. It is a good idea to have a short timeout between retries (around 100ms should be enough).

Too many "shoulds" here ...

I have amended an in-house script so that a single re-try takes place after a 100ms delay and I'm still getting the 501 error.

Do I need to extend the timeout, the number of retries or something else?

We were asked to give a concrete example of how this affects our apps, so here it is:

We do offer a migration service that enables customers to switch to draw.io from other diagramming tools without any barriers. Therefore we use a mass importer, which converts all existing diagrams and writes the draw.io macro and attachment to the page at once. So we we don't this service for one page, but up to thousands of times within a short time period.

These people want to see that we can deliver what we promise - which means to perform the migration within the 30 day trial period before they even think of becoming our customers. Now the the same people read our marketing collateral, trying it out and see that we can't deliver what we promise - so they move on because they don't believe in the solution we offer.

At the same time, we have three big customers that have complex migrations that we were in close contact with during the last months. They are now stuck between worlds because the migration doesn't work anymore. They can't go live with draw.io because the bulk importer was broken after they finished the evaluation and testing period.

The workaround just don't work out for us, there have been multiple examples already that prove we are not the only ones experiencing this, so I don't see the need of providing more evidence.

We are not talking of something that we would like to have. This is actually damaging our brand and our reputation. Not speaking of the money we lose each day this isn't fixed. At the same time we don't want to put the blame on Atlassian, because this is still a partnership. But the point is: Everybody who is on Confluence Server 7.x will not become a new customer, if they used existing diagramming apps before and like to switch.

If this situation will not be fixed soon, at least please take some time to provide a reliable workaround. Can our engineers get in contact with yours so we can provide a solution to our customers that is working? Thank you.

We also use PageManager.saveNewVersion() and one of our customer of Requirement Yogi was also affected by this issue. He was able to use the workaround of visiting each page before relaunching the process in Requirement Yogi.

Could we get an acknowledgement of the status within Atlassian please? I know a fix is a long way out, but is there any thinking about workarounds? Or, is the issue paused waiting for case studies to determine the impact to users?

This is causing us huge issues with our documentation space especially. And since Atlassian uses the same plugin we do (K15t's ScrollVersions), you would think they would also need this fixed. With the impact to numerous plugins, it should be clear that this is an urgent issue needing fixed.

I can confirm this bug still exists with version 7.2.0

Is there an API to refresh page?

That would have been my question as well

choeger added a comment - Yesterday

Currently, I manually refresh all pages within a given space and run the script.

@Carsten - How do you refresh pages? Is it sufficient to run /rest/api/content/id=<PAGEID> on all pages?

I manually refresh the pages. Will the following refresh a given page?

/rest/api/content/id=<PAGEID>

Is there an API to refresh pages?

Can Atlassian let us know please when the status of the ticket will change?

 After checking the [Atlassian bug fixing policy|https://confluence.atlassian.com/support/atlassian-bug-fixing-policy-201294573.html] I think we all agree we are surelly not under 'Gathering Impact' phase.

but more under the 

And we need the fix for the issue ASAP. One other question - under the bug version 7.2 is not mentioned. That means that if we upgrade to Confluence version 7.2 the issue is not present anymore? Is Atlassian guaranteeing this?

Thank you.

Currently, I manually refresh all pages within a given space and run the script.

How do you refresh pages? Is it sufficient to run /rest/api/content/id=<PAGEID> on all pages?

Currently, I manually refresh all pages within a given space and run the script. It sorts of works. It can get tedious if the number of pages that are being updated through the script run into something like 100+ of them. It can be even inefficient if we ask someone inexperienced to run the script. So if you resolve this issue through some kind of a patch, that would be great. Using REST APIs becomes beneficial only if it allows us to update/create a great number of pages at once without any obstacles. If it's one or two pages, we could do that manually.

Since there is no reliable workaround, what are the plans? Can we expect the status of this issue to be updated as well? 

I think there has been enough feedback and from all the comments it is clear that many app vendors suffer from this bug. Our apps require Confluence working reliably. Please fix it asap.

I completely agree with Daria Huber's comment above.

I think that the complexity of handling this problem should not be pushed to the client, in this case to all apps that want to create a new page version. Re-trying this multiple times with time delays does not sound like a very elegant way of handling this...

It should be transparently handled in the Confluence side:

1. You should either transparently guarantee the synchronized state within Confluence when apps call those methods that require that state (even at the cost of the extra 100 ms).
2. Or, offer a method the apps can explicitly call and which guarantees the synchronized state when it returns.

Good morning, thank you for updating the tickets priority accordingly. Will the status be updated as well? When can we expect a fix?

For version 7.1 the workaround is NOT working. Please increase the priority of the ticket to MAJOR. Thank you

For REST API, the best option would be to re-try the operation. As mentioned above, the publish/update attempt will trigger the reconciliation process if the page is un-synchronized. A re-try of the same request should succeed. It is a good idea to have a short timeout between retries (around 100ms should be enough).

This workaround does not seem to help. Implemented 5 retries with 1000ms sleep between each attempts, still no success in case it happens.

Additonally, there is a conflict between the updated status comment and the issue description.

In the issue description there is a workaround section that states:

"7.0.2 - Disabling/enabling collaborative editing clears the 'sync-rev-source' values and allows the calls to work. This method does not work for 7.1.0."

In the updated comment:

"In this situation, a workaround is to turn off collaborative editing for the time of migration. That will prevent synchronization checks and this error from occurring."

Please clarify the exact workaround position. Does that work for all 7.x confluence versions? Could the description and comment come into alignment regarding the workaround please?

Maksym, thanks for the additional information. You mention the ExternalChangesException in your comment, at runtime we get:

java.lang.NoClassDefFoundError: com/atlassian/confluence/internal/synchrony/ExternalChangesException

It compiles, just trips up when running.

https://docs.atlassian.com/atlassian-confluence/6.1.1/com/atlassian/confluence/internal/synchrony/ExternalChangesException.html

Suggests that this is a syncrony internal class. Is there a dependency we can add to be able to handle the exception? I hope we don't have to pull in the entire syncrony codebase to do that. Listening to the parent, UnsupportedOperationException, is non optimal, if the write is impossible for another reason we might retry a huge volume of calls with no idea that none of that could ever work.

Thank you for the update.  Where is that information that states that the workaround with turning off collaborative ediing is not working for version 7.1? We employed the suggested workarund, we turned off collaborative editing and the migration of Gliffy diagrams is still unsuccessful. This bug needs to be fixed, there is no workaround. Please increase the prirority to major, as if is bloking us completelly.

Well, the manual retry approach didn't work well for us. If one of the processes of our app failed, we restarted it again and again. But at some point, it was stuck and the reconciliation process was not initiated. We had to view the affected page (and sometimes even to refresh it multiple times) until we could start our app's process again. For me, it seems that the 100ms are not enough and will highly depend on the network response times and other circumstances.

Implementation of the ExternalChangesException handling would cause lots of changes in the code base for most of our products AND would add unnecessary complexity on our side, which would surely lead to more support load (not to mention the costs of the implementation) and unsatisfied customers. 

I hope this bug will be fixed very soon.

thnx for mentioning the delay of 100ms, changed it in our pipeline as i had a much longer pause defined between the api requests.

Thank you for all the insights on this issue and apologies for the inconvenience it is causing.

First, we would like to give a bit more context on why our API is behaving like this. Confluence has a service for managing collaborative editing on pages (Synchrony) and syncing these changes back to the database. To avoid data loss between this service and Confluence we need to ensure these two services are synced before making any changes to a page.

This error occurs when an update attempts to modify pages that have not yet been synchronized. Pages affected by this are identified by a “restored” value in the page content properties table. Such un-synchronization happens, when:

1. Collaborative editing is turned off and on again
2. Site/space is restored from back-up
3. Collaborative data is evicted (Confluence 7.0 release notes)
4. After running some upgrade tasks (for example, in Confluence 7.0, upgrade tasks were run to change data format)

Note: In 7.0+ this has become a more prominent issue because of data eviction and 7.0 upgrade tasks.

To restore these pages to a synchronized state, a reconciliation process is triggered lazily after a failed attempt to update one of these pages. This reconciliation process is expensive if it were to maintain constant synchronization between Confluence and Synchrony.

When end-users of Confluence access pages in this un-synchronized state reconciliation happens transparently in the browser and does not affect page viewing or editing. However, some extra work is required for REST/JAVA clients:

• For REST API, the best option would be to re-try the operation. As mentioned above, the publish/update attempt will trigger the reconciliation process if the page is un-synchronized. A re-try of the same request should succeed. It is a good idea to have a short timeout between retries (around 100ms should be enough).
• For JAVA API, similar steps apply: re-try the operation on ExternalChangesException. The challenge here is that a retry may need to happen in the new transaction. For example, if ExternalChangesException exception has crossed current transaction boundaries. This depends on operation logic, transaction management mechanism in the plugin, etc.

Not following the procedure above may result in your plugin or REST call failing to update pages.

For extreme circumstances, such as once-off data migration, the re-tries approach may be time-consuming because of the size of the dataset. In this situation, a workaround is to turn off collaborative editing for the time of migration. That will prevent synchronization checks and this error from occurring.

Data synchronization before update is a necessary process to support the data integrity of pages for every Confluence customer and as a result this issue is not something easily resolved. With this becoming a more prominent issue with recent releases of Confluence we are committed to assisting our developer community to find improved ways to update the content of pages. If this issue impacts you and the solutions above are not fit for your use case please let us know in the comments below. Feedback about your use cases and why the suggested approaches don’t work will help us to find the right solutions to address this issue.

Thanks,

The Confluence Development Team

Is this bug still considered to have Low priority Atlassian?

We've also got this issue with version nr 7.2.0 -  It seems this error appears only in relative new versions of confluence (7.x.x)

This is completely blocking us migrating from gliffy to lucidchart. I think there are hundreds of thousands more having the same issues, would be great to have them all here to vote and possibly raise the priority for that bug.

There is no workaround for 7.1 and 7.2. These are mission critical instances, nobody is going to try operations that risk database issues.

We are having the same issue, moving from Gliffy to Lucidchart. Please increase the priority to major and fix it ASAP. Thank you

There is no viable workaround. This is simply wrong in the issue description:

• Viewing the page is a nice workaround for a small number of affected pages but you can't tell customers to visit hundreds of pages and then figure out which apps might have ran into problems during their upgrade processes ... and then fix those in some way.
• Toggling the collaborative editing mode is not a workaround because it only works for 7.0.x and 7.3 is around the corner ...

This bug affects a core Confluence API that is used by many 3rd party apps and also by Atlassian plugins. I do not understand how this can be considered a minor bug.

Unfortunately, by Atlassian's own rules (https://confluence.atlassian.com/support/atlassian-bug-fixing-policy-201294573.html), because there appears to be a workaround, that allows them to put the severity as minor.

If there are use-cases where the workaround doesn't help, I suggest you create an issue on https://support.atlassian.com/ and make it clear that this is a major or critical problem for you. Hopefully, if enough customers start raising support issues rather than just commenting on here, the support agents might be able to persuade the powers-that-be that this is more serious than minor.

If we can get this reclassified as critical ("Users aren't able to perform their job function, and no workarounds are available.") ... "We'll generally fix critical bugs in the next maintenance release". Fingers crossed.

We want to migrate from Gliffy to draw.io but are unable to. This should indeed not be a minor bug.