Description
After Webhook events are sent there are some statistics recorded on the database to store the number of successful calls, as well as last success and last failure times.
Due to this bug https://ecosystem.atlassian.net/browse/AO-3395 the statistics may fail to update on Oracle databases
A stacktrace like the one below is logged in the application logs:
WARN [threadpool:thread-15] c.a.b.i.w.d.AsyncInvocationHistoryDao Failed to write webhook invocation data for <event> to the database com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library: Database: - name:Oracle - version:Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options - minor version:1 - major version:12 Driver: - name:Oracle JDBC driver - version:12.1.0.1.0 java.sql.SQLException: ORA-24816: Expanded non LONG bind data supplied after actual LONG or LOB column at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:102) at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.create(TenantAwareActiveObjects.java:271) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) at com.atlassian.bitbucket.internal.webhook.dao.AoInvocationHistoryDao.saveInvocation(AoInvocationHistoryDao.java:285) at com.atlassian.bitbucket.internal.webhook.dao.AsyncInvocationHistoryDao.maybeFlush(AsyncInvocationHistoryDao.java:233) at com.atlassian.bitbucket.internal.webhook.dao.AsyncInvocationHistoryDao.flushPending(AsyncInvocationHistoryDao.java:218) at com.atlassian.bitbucket.internal.webhook.dao.AsyncInvocationHistoryDao.flush(AsyncInvocationHistoryDao.java:179) at com.atlassian.bitbucket.internal.webhook.dao.AsyncInvocationHistoryDao.flushAndReschedule(AsyncInvocationHistoryDao.java:208) at com.atlassian.stash.internal.concurrent.DefaultTransferableStateManager$StateTransferringRunnable.run(DefaultTransferableStateManager.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.lang.Thread.run(Thread.java:748) ... 16 frames trimmed Caused by: java.sql.SQLException: ORA-24816: Expanded non LONG bind data supplied after actual LONG or LOB column at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450) at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1017) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:655) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:249) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:566) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:215) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:58) at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1075) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3820) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3897) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1361) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) at net.java.ao.db.OracleDatabaseProvider.executeInsertReturningKey(OracleDatabaseProvider.java:297) at net.java.ao.DatabaseProvider.insertReturningKey(DatabaseProvider.java:1861) at net.java.ao.EntityManager.create(EntityManager.java:366) at net.java.ao.EntityManager.create(EntityManager.java:399) at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:100) ... 19 common frames omitted
Webhook notifications should still be sent, even though the statistics are not updated correctly.
This problem can occur when using version 12.1.0 of the Oracle jdbc driver.
Workaround
Upgrading to version 12.2.0 of the Oracle jdbc driver seems to resolve the problem.