Navlinks plugin making too many connections to DB causing saturation

It has been reported that the navlinks plugin seems to be running database queries to lookup information about certain application links very often (over 4 times per second). This seems to be opening up new connections to the database pool which then becomes saturated to the point of failure - java.lang.InterruptedException.

This seems to be related to CONFDEV-17389, however it is unsure whether the root cause was fixed for that.

This can cause performance issues and affect the page load times. In a extreme case it could even bring down the server and cause database connection pool starvation.

Errors in the logs:

ERROR [Navlink Plugin Executor:thread-3] [sf.hibernate.util.JDBCExceptionReporter] logExceptions An SQLException was provoked by the following failure: java.lang.InterruptedException

WARN [Navlink Plugin Executor:thread-4] [mchange.v2.resourcepool.BasicResourcePool] prelimCheckoutResource com.mchange.v2.resourcepool.BasicResourcePool@ -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have either interrupted the Thread attempting checkout!
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
	at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
	at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
	at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
	at net.sf.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:33)
	at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:292)
	at net.sf.hibernate.impl.SessionImpl.connect(SessionImpl.java:3388)
	at net.sf.hibernate.impl.SessionImpl.connection(SessionImpl.java:3348)
	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
..

Workarounds

The below are mitigation steps that could be taken to minimize the impact of this issue.

• Reduce the number of threads in Navlink Plugin Executor (which defaults to 12), to 4. This requires the below JVM startup parameter to be passed, in the manner as described in this page:

  -Dnavlink.executor.threadpool=4 
  

• Review Confluence Admin > Application Navigator, and remove invalid applications (if any). E.g. any that are offline or no longer in use
• Increase the database connection pool. See Step 4 in the Resolution of this KB article

Disabling the plugin

If the above workarounds aren't working for you, it may be necessary to disable the plugin:

1. Shut down Confluence
2. Perform a full database backup
3. Obtain the current plugin state map:

   select BANDANAVALUE from BANDANA where BANDANAKEY = 'plugin.manager.state.Map'

If it's an empty map (<map/>) then you can use the following value to update the table:

<map>
  <entry>
    <string>com.atlassian.confluence.plugins.confluence-nav-links</string>
    <boolean>false</boolean>
  </entry>
</map>

Otherwise, you'll need to add the entry <entry> to the existing map, and then save that.

UPDATE BANDANA SET BANDANAVALUE = 'xxx' WHERE BANDANAKEY = 'plugin.manager.state.Map'

Replace xxx with your updated map. For example, here's the change I made in my local instance:

UPDATE BANDANA SET BANDANAVALUE = '<map>
  <entry>
    <string>com.atlassian.confluence.plugins.confluence-nav-links</string>
    <boolean>false</boolean>
  </entry>
</map>' WHERE BANDANAKEY = 'plugin.manager.state.Map'

After making the change, restart Confluence. You'll notice that the application switcher (The three lines in the top left hand corner of the screen) are now gone - that's the application switcher that's been disabled as a result of disabling the Navlinks plugin.

To revert the change, either update that value with <map/> again, or restore the database backup.

Excessive Logging

You may receive a lot of logging stating that the Navlinks plugin has been disabled. You can add the following statement to log4j.properties to suppress the message (as it's not particularly helpful). Please note that Confluence will need to be restarted to pick up this logging change:

log4j.logger.com.atlassian.confluence.plugins.soy.VelocityFriendlySoyTemplateRenderer=ERROR