[CONFSERVER-9689] Dashboard error on Extranet caused by duplicate attachment data retrieval

Type: Bug
Resolution: Duplicate
Priority: Medium
Fix Version/s: None
Affects Version/s: 2.7
Component/s: None
Labels:
- affects-server
- editor

Bug Fix Policy:
View Atlassian Server bug fix policy

The error on the dashboard is simply a transaction rollback:

org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:626)
at com.atlassian.xwork.interceptors.XWorkTransactionInterceptor.intercept(XWorkTransactionInterceptor.java:132)

The underlying cause is this:

2007-10-10 18:29:27,981 ERROR [TP-Processor18] [renderer.v2.components.MacroRendererComponent] processMacro Unexpected error formatting macro: recently-updated
– url: /dashboard.action | userName: rosie@atlassian.com | action: dashboard
java.lang.RuntimeException: Multiple AttachmentData objects were returned when only one was expected
at com.atlassian.confluence.pages.persistence.dao.hibernate.HibernateAttachmentDataDao.getAttachmentDataForAttachment(HibernateAttachmentDataDao.java:66)
at com.atlassian.confluence.pages.persistence.dao.hibernate.HibernateAttachmentDataDao.getDataForAttachment(HibernateAttachmentDataDao.java:76)
at com.atlassian.confluence.pages.persistence.dao.hibernate.AbstractHibernateAttachmentDao.getAttachmentData(AbstractHibernateAttachmentDao.java:109)
at com.atlassian.confluence.pages.DefaultAttachmentManager.getAttachmentData(DefaultAttachmentManager.java:33)
at com.atlassian.confluence.pages.DelegatorAttachmentManager.getAttachmentData(DelegatorAttachmentManager.java:77)
at sun.reflect.GeneratedMethodAccessor286.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy15.getAttachmentData(Unknown Source)
at com.atlassian.confluence.util.DefaultThumbnailInfoFactory.getThumbnailInfo(DefaultThumbnailInfoFactory.java:24)
at com.atlassian.confluence.renderer.radeox.macros.RecentlyUpdatedContentMacro.addThumbnailData(RecentlyUpdatedContentMacro.java:303)
at com.atlassian.confluence.renderer.radeox.macros.RecentlyUpdatedContentMacro.getChangeSets(RecentlyUpdatedContentMacro.java:253)
at com.atlassian.confluence.renderer.radeox.macros.RecentlyUpdatedContentMacro.getHtml(RecentlyUpdatedContentMacro.java:206)
at com.atlassian.confluence.renderer.radeox.macros.AbstractHtmlGeneratingMacro.execute(AbstractHtmlGeneratingMacro.java:42)
at com.atlassian.renderer.macro.RadeoxCompatibilityMacro.execute(RadeoxCompatibilityMacro.java:149)
at com.atlassian.confluence.renderer.v2.macros.RadeoxCompatibilityMacro.execute(RadeoxCompatibilityMacro.java:27)
at com.atlassian.renderer.v2.macro.ResourceAwareMacroDecorator.execute(ResourceAwareMacroDecorator.java:45)
at com.atlassian.renderer.v2.components.MacroRendererComponent.processMacro(MacroRendererComponent.java:254)
at com.atlassian.renderer.v2.components.MacroRendererComponent.makeMacro(MacroRendererComponent.java:150)
at com.atlassian.renderer.v2.components.WikiContentRendererHandler.handleMacro(WikiContentRendererHandler.java:18)
at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:126)
at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:114)
at com.atlassian.renderer.v2.WikiMarkupParser.handlePotentialMacro(WikiMarkupParser.java:99)
at com.atlassian.renderer.v2.WikiMarkupParser.parse(WikiMarkupParser.java:60)
at com.atlassian.renderer.v2.components.MacroRendererComponent.render(MacroRendererComponent.java:46)
at com.atlassian.renderer.v2.V2Renderer.render(V2Renderer.java:54)
at com.atlassian.renderer.v2.V2RendererFacade.convertWikiToXHtml(V2RendererFacade.java:57)
at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:20)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy37.convertWikiToXHtml(Unknown Source)
at com.atlassian.confluence.renderer.DefaultWikiStyleRenderer.convertWikiToXHtml(DefaultWikiStyleRenderer.java:19)
at com.atlassian.confluence.themes.GlobalHelper.renderConfluenceMacro(GlobalHelper.java:149)
at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:260)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:207)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:250)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:94)
at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:109)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:271)
at org.apache.velocity.Template.merge(Template.java:296)
at com.atlassian.confluence.util.velocity.VelocityUtils.getRenderedTemplateWithoutSwallowingErrors(VelocityUtils.java:53)
at com.atlassian.confluence.setup.velocity.ApplyDecoratorDirective.render(ApplyDecoratorDirective.java:229)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:153)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:271)
at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:215)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:153)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:94)
at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:109)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:271)
at org.apache.velocity.Template.merge(Template.java:296)
at com.atlassian.confluence.util.velocity.VelocityUtils.getRenderedTemplateWithoutSwallowingErrors(VelocityUtils.java:53)
at com.atlassian.confluence.setup.velocity.ApplyDecoratorDirective.render(ApplyDecoratorDirective.java:229)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:153)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:271)
at org.apache.velocity.Template.merge(Template.java:296)
at com.opensymphony.webwork.dispatcher.VelocityResult.doExecute(VelocityResult.java:91)
at com.atlassian.xwork.results.ProfiledVelocityResult.doExecute(ProfiledVelocityResult.java:21)
at com.opensymphony.webwork.dispatcher.WebWorkResultSupport.execute(WebWorkResultSupport.java:116)
...

duplicates

CONFSERVER-7882 Multiple AttachmentData objects were returned when only one was expected

Closed

is caused by

CONFSERVER-10662 Confluence allows multiple versions of the same attachment to be added

Closed

is duplicated by

CONFSERVER-30596 Certain Special Character In The Change Comment are Stripped In The Recently-Updated-Dashboard Macro

Closed

relates to

CONFSERVER-9856 Resource is locked by another thread or member, causing pages to crash

Closed

Don Willis added a comment - 04/Nov/2011 12:39 AM

The initial description of this bug is covered by ~~CONF-7882~~ the various other webdav plugin discussions are likely to be completely obsolete by now, since the Webdav plugin has been rewritten in the meantime.

Don Willis added a comment - 04/Nov/2011 12:39 AM The initial description of this bug is covered by CONF-7882 the various other webdav plugin discussions are likely to be completely obsolete by now, since the Webdav plugin has been rewritten in the meantime.

Matt Ryall added a comment - 03/Mar/2009 11:44 PM

Steve is now using XML-RPC to upload his attachments, which prevent the zero-byte files being created and also don't have this problem with duplicates.

Matt Ryall added a comment - 03/Mar/2009 11:44 PM Steve is now using XML-RPC to upload his attachments, which prevent the zero-byte files being created and also don't have this problem with duplicates.

Paul Curren added a comment - 16/Jan/2008 6:37 AM

This fix did not make it into 2.7.1. It will be addressed in a later release.

Paul Curren added a comment - 16/Jan/2008 6:37 AM This fix did not make it into 2.7.1. It will be addressed in a later release.

David Peterson added a comment - 17/Dec/2007 1:47 AM

What exactly is the status of this issue? Has it been tested with the current trunk? I'm not sure if WebDAV was released after I went on leave...

David Peterson added a comment - 17/Dec/2007 1:47 AM What exactly is the status of this issue? Has it been tested with the current trunk? I'm not sure if WebDAV was released after I went on leave...

Paul Curren added a comment - 14/Dec/2007 4:38 AM - edited

I think the duplicate versions handling seems to have been fixed, or for whatever reason the recently-updated macro is no longer falling down.

However, there are very related issues (or the same issues with a different exception) is now appearing regularly in the EAC logs.
(This is coming from the same attachments that were causing the previously discussed problem.)

2007-12-13 20:50:21,323 ERROR [TP-Processor8] [[Catalina].[localhost].[/].[file-server]] invoke Servlet.service() for servlet file-server threw exception
java.lang.RuntimeException: java.io.FileNotFoundException: Thumbnail file for attachment graph-internode.png is '/opt/j2ee/domains/atlassian.com/extranet/
webapps/atlassian-confluence/data/thumbnails/69518087748061382/thumb_graph-internode.png'. But this file does not exist.
        at com.atlassian.confluence.importexport.resource.ThumbnailDownloadResourceReader.getStreamForReading(ThumbnailDownloadResourceReader.java:57)
        at com.atlassian.confluence.servlet.download.AttachmentDownload.getStreamForDownload(AttachmentDownload.java:94)
        at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload$StreamResultCallback.doInTransaction(ServeAfterTransactionDownload.java
:128)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127)
        at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.getStreamInTransaction(ServeAfterTransactionDownload.java:40)
        at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.serveFile(ServeAfterTransactionDownload.java:46)
        at com.atlassian.plugin.servlet.BaseFileServerServlet.doGet(BaseFileServerServlet.java:77)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

and

2007-12-13 20:50:21,312 ERROR [TP-Processor8] [core.util.thumbnail.Thumber] retrieveOrCreateThumbNail Unable to create thumbnail image of attachment with 
id:665944169
 -- url: /download/thumbnails/69518087748061382/graph-internode.png | userName: pcurren
java.lang.NullPointerException
        at com.atlassian.core.util.thumbnail.Thumber.scaleImage(Thumber.java:156)
        at com.atlassian.core.util.thumbnail.Thumber.createThumbnail(Thumber.java:170)
        at com.atlassian.core.util.thumbnail.Thumber.retrieveOrCreateThumbNail(Thumber.java:131)
        at com.atlassian.confluence.pages.thumbnail.DefaultThumbnailManager.getThumbnail(DefaultThumbnailManager.java:75)
        at com.atlassian.confluence.importexport.resource.ThumbnailDownloadResourceManager.getResourceReader(ThumbnailDownloadResourceManager.java:44)
        at com.atlassian.confluence.importexport.resource.DelegatorDownloadResourceManager.getResourceReader(DelegatorDownloadResourceManager.java:40)
        at com.atlassian.confluence.servlet.download.AttachmentDownload.getStreamForDownload(AttachmentDownload.java:54)
        at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload$StreamResultCallback.doInTransaction(ServeAfterTransactionDownload.java
:128)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127)
        at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.getStreamInTransaction(ServeAfterTransactionDownload.java:40)
        at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.serveFile(ServeAfterTransactionDownload.java:46)
        at com.atlassian.plugin.servlet.BaseFileServerServlet.doGet(BaseFileServerServlet.java:77)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:49)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

Paul Curren added a comment - 14/Dec/2007 4:38 AM - edited I think the duplicate versions handling seems to have been fixed, or for whatever reason the recently-updated macro is no longer falling down. However, there are very related issues (or the same issues with a different exception) is now appearing regularly in the EAC logs. (This is coming from the same attachments that were causing the previously discussed problem.) 2007-12-13 20:50:21,323 ERROR [TP-Processor8] [[Catalina].[localhost].[/].[file-server]] invoke Servlet.service() for servlet file-server threw exception java.lang.RuntimeException: java.io.FileNotFoundException: Thumbnail file for attachment graph-internode.png is '/opt/j2ee/domains/atlassian.com/extranet/ webapps/atlassian-confluence/data/thumbnails/69518087748061382/thumb_graph-internode.png'. But this file does not exist. at com.atlassian.confluence.importexport.resource.ThumbnailDownloadResourceReader.getStreamForReading(ThumbnailDownloadResourceReader.java:57) at com.atlassian.confluence.servlet.download.AttachmentDownload.getStreamForDownload(AttachmentDownload.java:94) at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload$StreamResultCallback.doInTransaction(ServeAfterTransactionDownload.java :128) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127) at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.getStreamInTransaction(ServeAfterTransactionDownload.java:40) at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.serveFile(ServeAfterTransactionDownload.java:46) at com.atlassian.plugin.servlet.BaseFileServerServlet.doGet(BaseFileServerServlet.java:77) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) and 2007-12-13 20:50:21,312 ERROR [TP-Processor8] [core.util.thumbnail.Thumber] retrieveOrCreateThumbNail Unable to create thumbnail image of attachment with id:665944169 -- url: /download/thumbnails/69518087748061382/graph-internode.png | userName: pcurren java.lang.NullPointerException at com.atlassian.core.util.thumbnail.Thumber.scaleImage(Thumber.java:156) at com.atlassian.core.util.thumbnail.Thumber.createThumbnail(Thumber.java:170) at com.atlassian.core.util.thumbnail.Thumber.retrieveOrCreateThumbNail(Thumber.java:131) at com.atlassian.confluence.pages.thumbnail.DefaultThumbnailManager.getThumbnail(DefaultThumbnailManager.java:75) at com.atlassian.confluence.importexport.resource.ThumbnailDownloadResourceManager.getResourceReader(ThumbnailDownloadResourceManager.java:44) at com.atlassian.confluence.importexport.resource.DelegatorDownloadResourceManager.getResourceReader(DelegatorDownloadResourceManager.java:40) at com.atlassian.confluence.servlet.download.AttachmentDownload.getStreamForDownload(AttachmentDownload.java:54) at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload$StreamResultCallback.doInTransaction(ServeAfterTransactionDownload.java :128) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127) at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.getStreamInTransaction(ServeAfterTransactionDownload.java:40) at com.atlassian.confluence.servlet.download.ServeAfterTransactionDownload.serveFile(ServeAfterTransactionDownload.java:46) at com.atlassian.plugin.servlet.BaseFileServerServlet.doGet(BaseFileServerServlet.java:77) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:192) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

Matt Ryall added a comment - 03/Dec/2007 11:20 PM

Is the WebDAV plugin doing its uploads through one of Confluence's managers?

If so, the onus is on Confluence not to allow its data integrity to be compromised by operations which can be done through the manager layer. The attachment manager (or whatever it is) should not allow duplicate attachments to be inserted in the database. We should be strict when uploaded, and ensure these dodgy client uploads fail, so that operations in Confluence later can work with consistent data.

We should also not fail operations in Confluence if there's a duplicate in there. It probably warrants a WARN-level log message, and just taking the first result.

Matt Ryall added a comment - 03/Dec/2007 11:20 PM Is the WebDAV plugin doing its uploads through one of Confluence's managers? If so, the onus is on Confluence not to allow its data integrity to be compromised by operations which can be done through the manager layer. The attachment manager (or whatever it is) should not allow duplicate attachments to be inserted in the database. We should be strict when uploaded, and ensure these dodgy client uploads fail, so that operations in Confluence later can work with consistent data. We should also not fail operations in Confluence if there's a duplicate in there. It probably warrants a WARN-level log message, and just taking the first result.

Paul Curren added a comment - 26/Oct/2007 12:39 AM

Some info on the WebDav client being used to upload the 'troubled' attachments.

The client is cadaver: http://www.webdav.org/cadaver/

The upload is wrapped in a python script, code here: https://svn.atlassian.com/privateeye/browse/private/atlassian/sysadmin/network-graphs

Paul Curren added a comment - 26/Oct/2007 12:39 AM Some info on the WebDav client being used to upload the 'troubled' attachments. The client is cadaver: http://www.webdav.org/cadaver/ The upload is wrapped in a python script, code here: https://svn.atlassian.com/privateeye/browse/private/atlassian/sysadmin/network-graphs

Matt Ryall added a comment - 12/Oct/2007 12:14 AM

Found the affected attachments. They're on this page: http://extranet.atlassian.com/display/SYSADMIN/Home

They were uploaded via WebDAV. Looks like there might be some issues with this method of uploading. The attachments have duplicates with the same version number.

Matt Ryall added a comment - 12/Oct/2007 12:14 AM Found the affected attachments. They're on this page: http://extranet.atlassian.com/display/SYSADMIN/Home They were uploaded via WebDAV. Looks like there might be some issues with this method of uploading. The attachments have duplicates with the same version number.

Don Willis added a comment - 11/Oct/2007 2:26 AM

I reliably get this error if I go to the dashboard with "?spacesSelectedTab=my". Any other spacesSelectedTab is fine.

Don Willis added a comment - 11/Oct/2007 2:26 AM I reliably get this error if I go to the dashboard with "?spacesSelectedTab=my". Any other spacesSelectedTab is fine.

Matt Ryall added a comment - 11/Oct/2007 1:58 AM

I've added better logging which includes the affected attachment. I couldn't see any duplicates in the database, and the attachment data doesn't appear to have a cache which could be including invalid data.

We'll hopefully get this patch to Extranet soon, although the affected page/attachment seems to have moved off the recently updated list so it's no longer hitting us so hard.

Matt Ryall added a comment - 11/Oct/2007 1:58 AM I've added better logging which includes the affected attachment. I couldn't see any duplicates in the database, and the attachment data doesn't appear to have a cache which could be including invalid data. We'll hopefully get this patch to Extranet soon, although the affected page/attachment seems to have moved off the recently updated list so it's no longer hitting us so hard.

Assignee:: Don Willis

Reporter:: Matt Ryall

Affected customers:: 1 This affects my team

Watchers:: 4 Start watching this issue

Created:: 11/Oct/2007 1:36 AM

Updated:: 11/Oct/2018 8:48 AM

Resolved:: 04/Nov/2011 12:39 AM

Details

Description

Attachments

Issue Links

Forms

Activity

Collapse comment: Don Willis added a comment - 04/Nov/2011 12:39 AM

Expand comment: Don Willis added a comment - 04/Nov/2011 12:39 AM

Collapse comment: Matt Ryall added a comment - 03/Mar/2009 11:44 PM

Expand comment: Matt Ryall added a comment - 03/Mar/2009 11:44 PM

Collapse comment: Paul Curren added a comment - 16/Jan/2008 6:37 AM

Expand comment: Paul Curren added a comment - 16/Jan/2008 6:37 AM

Collapse comment: David Peterson added a comment - 17/Dec/2007 1:47 AM

Expand comment: David Peterson added a comment - 17/Dec/2007 1:47 AM

Collapse comment: Paul Curren added a comment - 14/Dec/2007 4:38 AM, Edited by Paul Curren - 14/Dec/2007 4:45 AM

Expand comment: Paul Curren added a comment - 14/Dec/2007 4:38 AM, Edited by Paul Curren - 14/Dec/2007 4:45 AM

Collapse comment: Matt Ryall added a comment - 03/Dec/2007 11:20 PM

Expand comment: Matt Ryall added a comment - 03/Dec/2007 11:20 PM

Collapse comment: Paul Curren added a comment - 26/Oct/2007 12:39 AM

Expand comment: Paul Curren added a comment - 26/Oct/2007 12:39 AM

Collapse comment: Matt Ryall added a comment - 12/Oct/2007 12:14 AM

Expand comment: Matt Ryall added a comment - 12/Oct/2007 12:14 AM

Collapse comment: Don Willis added a comment - 11/Oct/2007 2:26 AM

Expand comment: Don Willis added a comment - 11/Oct/2007 2:26 AM

Collapse comment: Matt Ryall added a comment - 11/Oct/2007 1:58 AM

Expand comment: Matt Ryall added a comment - 11/Oct/2007 1:58 AM

People

Dates