Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-8632

NullPointerException from Hazelcast operations in Data Center installation after failed partition migration

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Medium
    • 4.7.1
    • 4.5.2
    • Data Center, Enterprise
    • None

    Description

      Symptoms

      • 1 node is shutdown in a Data Center installation
      • partition migration on shutdown fails:
      2016-04-14 18:20:33,324 WARN  [hz.hazelcast.migration]  c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Failed migration from Member [<ip2>]:5701 for com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Add
      ress[<ip1>]:5701, valid=true, processing=false}
      com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster!
              at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry$OnMemberLeftTask.run(InvocationRegistry.java:398) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
              at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
              at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at ------ End remote and begin local stack-trace ------.(Unknown Source) ~[na:na]
              at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.executeMigrateOperation(InternalPartitionServiceImpl.java:1826) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.run(InternalPartitionServiceImpl.java:1795) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.processTask(InternalPartitionServiceImpl.java:1931) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:1907) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:1884) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
              ... 1 frame trimmed
      2016-04-14 18:20:33,324 WARN  [hz.hazelcast.migration]  c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Migration failed: com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Address[<ip1>]:5701, valid=true
      , processing=false}
      

      <23 hours pass>

      • Error 500 while error the projects page
      • Errors in the logs with the following stacktrace:
      ...
      Caused by: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.serial.PROJECT
      	at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:278) ~[atlassian-cache-hazelcast-2.11.0.jar:na]
      	at com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:107) ~[atlassian-cache-hazelcast-2.11.0.jar:na]
      	at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorateUrl(VersionAwareAvatarUrlDecorator.java:62) ~[bitbucket-service-impl-4.5.1.jar:na]
      	at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorate(VersionAwareAvatarUrlDecorator.java:38) ~[bitbucket-service-impl-4.5.1.jar:na]
      	at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:429) ~[bitbucket-service-impl-4.5.1.jar:na]
      	at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:418) ~[bitbucket-service-impl-4.5.1.jar:na]
      	at com.atlassian.stash.internal.project.InternalNormalProject.accept(InternalNormalProject.java:37) ~[bitbucket-model-4.5.1.jar:na]
      	at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:219) ~[bitbucket-service-impl-4.5.1.jar:na]
      	at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na]
      	at com.atlassian.bitbucket.util.RequestLocalCache.get(RequestLocalCache.java:62) ~[bitbucket-util-4.5.1.jar:na]
      	at com.atlassian.stash.internal.avatar.ConfigurableAvatarService.getUrlForProject(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na]
      	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) ~[atlassian-plugins-core-4.1.1.jar:na]
      	at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.getUrl(AvatarUrlFunction.java:78) ~[classes/:na]
      	at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:67) ~[classes/:na]
      	at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:23) ~[classes/:na]
      	at com.atlassian.soy.impl.modules.SoyJavaFunctionAdapter.computeForJava(SoyJavaFunctionAdapter.java:34) ~[na:na]
      	at com.atlassian.soy.impl.modules.CompositeFunctionAdaptor.computeForJava(CompositeFunctionAdaptor.java:38) ~[na:na]
      	at com.google.template.soy.sharedpasses.render.EvalVisitor.computeFunctionHelper(EvalVisitor.java:670) ~[na:na]
      	... 132 common frames omitted
      

      Attachments

        Issue Links

          Activity

            People

              cszmajda Cristan Szmajda (Inactive)
              mheemskerk Michael Heemskerk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: