Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-26381

The loading of plugins symultaniously seems to be maxing out on Jira on Demand

    XMLWordPrintable

Details

    Description

      When starting up an instance from scratch (no db or home) in onDemand. When all the plugins are simultaneously being unzipped by felix, if you monitor the JIRA instance there is plenty of heap and permgen it seems to be happening at the c/malloc level. Exceptions due to memory exhaustion happen. Subsiquent startups are screwed as the db is not nicely configured.

      The first exception is something along the lines of (a few times)

      java.io.FileNotFoundException: /data/jirastudio/jira/home/plugins/.osgi-plugins/felix/felix-cache/bundle90/version0.0/bundle.jar-embedded/META-INF/lib/sisyphus-scanner-tools-2.jar (Cannot allocate memory)
      @400000004ececc382acf01dc       at java.io.FileOutputStream.open(Native Method)
      @400000004ececc382acf01dc       at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
      @400000004ececc382acf211c       at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
      @400000004ececc382acf211c       at org.apache.felix.framework.util.SecureAction.getFileOutputStream(SecureAction.java:461)
      @400000004ececc382acf2504       at org.apache.felix.framework.cache.BundleCache.copyStreamToFile(BundleCache.java:243)
      @400000004ececc382acf28ec       at org.apache.felix.framework.cache.JarContent.extractEmbeddedJar(JarContent.java:460)
      @400000004ececc382acf2cd4       at org.apache.felix.framework.cache.JarContent.getEntryAsContent(JarContent.java:254)
      @400000004ececc382acf30bc       at org.apache.felix.framework.ModuleImpl.calculateContentPath(ModuleImpl.java:584)
      @400000004ececc382acf30bc       at org.apache.felix.framework.ModuleImpl.initializeContentPath(ModuleImpl.java:535)
      @400000004ececc382acf405c       at org.apache.felix.framework.ModuleImpl.getContentPath(ModuleImpl.java:522)
      @400000004ececc382acf405c       at org.apache.felix.framework.ModuleImpl.access$600(ModuleImpl.java:73)
      @400000004ececc382acf4444       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.findClass(ModuleImpl.java:1732)
      @400000004ececc382acf4444       at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:716)
      @400000004ececc382acf4c14       at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73)
      @400000004ececc382acf4c14       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690)
      @400000004ececc382acf4ffc       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      @400000004ececc382acf4ffc       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java)
      @400000004ececc382acf57cc       at org.apache.felix.framework.ModuleImpl.getClassByDelegation(ModuleImpl.java:634)
      @400000004ececc382acf57cc       at org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:541)
      @400000004ececc382acf5bb4       at org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:278)
      @400000004ececc382acf5bb4       at org.apache.felix.framework.Felix.getServiceReferences(Felix.java:2920)
      @400000004ececc382acf6f3c       at org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:2970)
      @400000004ececc382acf770c       at org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:309)
      @400000004ececc382acf7af4       at org.osgi.util.tracker.ServiceTracker.getInitialReferences(ServiceTracker.java:362)
      @400000004ececc382acf7edc       at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:311)
      @400000004ececc382acf7edc       at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:273)
      @400000004ececc382acf82c4       at com.atlassian.plugin.osgi.factory.OsgiPluginInstalledHelper.onEnable(OsgiPluginInstalledHelper.java:92)
      @400000004ececc382acf86ac       at com.atlassian.plugin.osgi.factory.OsgiPlugin.enableInternal(OsgiPlugin.java:404)
      @400000004ececc382acf8a94       at com.atlassian.plugin.impl.AbstractPlugin.enable(AbstractPlugin.java:237)
      @400000004ececc382acf8a94       at com.atlassian.plugin.impl.AbstractDelegatingPlugin.enable(AbstractDelegatingPlugin.java:213)
      @400000004ececc382acf8e7c       at com.atlassian.plugin.manager.PluginEnabler.actualEnable(PluginEnabler.java:114)
      @400000004ececc382acf9264       at com.atlassian.plugin.manager.PluginEnabler.enable(PluginEnabler.java:99)
      @400000004ececc382acf9264       at com.atlassian.plugin.manager.DefaultPluginManager.addPlugins(DefaultPluginManager.java:682)
      @400000004ececc382acf964c       at com.atlassian.plugin.manager.DefaultPluginManager.init(DefaultPluginManager.java:156)
      @400000004ececc382acf9e1c       at com.atlassian.jira.plugin.JiraPluginManager.start(JiraPluginManager.java:61)
      @400000004ececc382acfa204       at com.atlassian.jira.ComponentManager$PluginSystem.start(ComponentManager.java:676)
      @400000004ececc382acfa5ec       at com.atlassian.jira.ComponentManager.startPluginSystem(ComponentManager.java:242)
      @400000004ececc382acfa5ec       at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:235)
      @400000004ececc382acfadbc       at com.atlassian.jira.ComponentManager.start(ComponentManager.java:220)
      @400000004ececc382acfadbc       at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:23)
      @400000004ececc382acfb1a4       at com.atlassian.jira.startup.DefaultJiraLauncher$3.run(DefaultJiraLauncher.java:101)
      @400000004ececc382acfb1a4       at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:250)
      @400000004ececc382acfb974       at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:149)
      @400000004ececc382acfb974       at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:94)
      @400000004ececc382acfc52c       at com.atlassian.jira.startup.DefaultJiraLauncher.access$100(DefaultJiraLauncher.java:24)
      @400000004ececc382acfc52c       at com.atlassian.jira.startup.DefaultJiraLauncher$1.run(DefaultJiraLauncher.java:61)
      @400000004ececc382acfc914       at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:33)
      @400000004ececc382acfc914       at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:56)
      @400000004ececc382acfd0e4       at com.atlassian.jira.startup.LauncherContextListener$1.create(LauncherContextListener.java:68)
      @400000004ececc382acfd0e4       at com.atlassian.jira.startup.LauncherContextListener$1.create(LauncherContextListener.java:63)
      @400000004ececc382acfd4cc       at com.atlassian.multitenant.impl.MultiTenantComponentMapImpl.get(MultiTenantComponentMapImpl.java:121)
      @400000004ececc382acfd8b4       at com.atlassian.multitenant.impl.MultiTenantComponentMapImpl.onTenantStart(MultiTenantComponentMapImpl.java:165)
      @400000004ececc382acfdc9c       at com.atlassian.multitenant.impl.DefaultMultiTenantManager$1.consume(DefaultMultiTenantManager.java:134)
      @400000004ececc382acfe084       at com.atlassian.multitenant.impl.DefaultMultiTenantManager$1.consume(DefaultMultiTenantManager.java:131)
      @400000004ececc382acfe084       at com.atlassian.multitenant.impl.DefaultMultiTenantManager.runForEachListener(DefaultMultiTenantManager.java:256)
      @400000004ececc382acff024       at com.atlassian.multitenant.impl.DefaultMultiTenantManager.startTenant(DefaultMultiTenantManager.java:130)
      @400000004ececc382acff40c       at com.atlassian.multitenant.impl.DefaultMultiTenantManager.startAll(DefaultMultiTenantManager.java:203)
      @400000004ececc382acff7f4       at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:96)
      

      then

      Unable to extract embedded JAR file.
      @400000004ececc382dbe710c java.io.IOException: Unable to create embedded JAR directory.
      @400000004ececc382dbe710c       at org.apache.felix.framework.cache.JarContent.extractEmbeddedJar(JarContent.java:449)
      @400000004ececc382dbe74f4       at org.apache.felix.framework.cache.JarContent.getEntryAsContent(JarContent.java:254)
      @400000004ececc382dbe74f4       at org.apache.felix.framework.ModuleImpl.calculateContentPath(ModuleImpl.java:584)
      @400000004ececc382dbe904c       at org.apache.felix.framework.ModuleImpl.initializeContentPath(ModuleImpl.java:535)
      @400000004ececc382dbe904c       at org.apache.felix.framework.ModuleImpl.getContentPath(ModuleImpl.java:522)
      @400000004ececc382dbe9434       at org.apache.felix.framework.ModuleImpl.access$600(ModuleImpl.java:73)
      @400000004ececc382dbe9434       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.findClass(ModuleImpl.java:1732)
      @400000004ececc382dbea7bc       at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:716)
      @400000004ececc382dbea7bc       at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73)
      @400000004ececc382dbeaba4       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690)
      @400000004ececc382dbeb75c       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      @400000004ececc382dbeb75c       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java)
      @400000004ececc382dbebb44       at org.apache.felix.framework.ModuleImpl.getClassByDelegation(ModuleImpl.java:634)
      @400000004ececc382dbebb44       at org.apache.felix.framework.resolver.WireImpl.getClass(WireImpl.java:99)
      @400000004ececc382dbec314       at org.apache.felix.framework.ModuleImpl.searchImports(ModuleImpl.java:1345)
      @400000004ececc382dbec314       at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:711)
      @400000004ececc382dbec6fc       at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73)
      @400000004ececc382dbec6fc       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690)
      @400000004ececc382dbececc       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      @400000004ececc382dbececc       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java)
      @400000004ececc382dbed2b4       at org.apache.felix.framework.ModuleImpl.getClassByDelegation(ModuleImpl.java:634)
      @400000004ececc382dbed2b4       at org.apache.felix.framework.Felix.loadBundleClass(Felix.java:1594)
      @400000004ececc382dbed69c       at org.apache.felix.framework.BundleImpl.loadClass(BundleImpl.java:887)
      @400000004ececc382dbede6c       at org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:99)
      @400000004ececc382dbee254       at org.springframework.osgi.util.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:156)
      @400000004ececc382dbee63c       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      @400000004ececc382dbee63c       at org.springframework.util.ClassUtils.forName(ClassUtils.java:211)
      @400000004ececc382dbeea24       at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:230)
      @400000004ececc382dbeee0c       at org.springframework.beans.propertyeditors.ClassArrayEditor.setAsText(ClassArrayEditor.java:66)
      @400000004ececc382dbef1f4       at org.springframework.beans.TypeConverterDelegate.doConvertTextValue(TypeConverterDelegate.java:382)
      @400000004ececc382dbef1f4       at org.springframework.beans.TypeConverterDelegate.doConvertValue(TypeConverterDelegate.java:358)
      @400000004ececc382dbef5dc       at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:173)
      @400000004ececc382dbef9c4       at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:138)
      @400000004ececc382dbefdac       at org.springframework.beans.BeanWrapperImpl.convertForProperty(BeanWrapperImpl.java:386)
      @400000004ececc382dbf057c       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.convertForProperty(AbstractAutowireCapableBeanFactory.java:1289)
      @400000004ececc382dbf0964       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1250)
      @400000004ececc382dbf0964       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1010)
      

      then

      SEVERE: Exception sending context initialized event to listener instance of class com.atlassian.jira.startup.LauncherContextListener
      @400000004ececc382dd7c184 java.lang.OutOfMemoryError
      @400000004ececc382dd7c184       at java.util.zip.ZipFile.open(Native Method)
      @400000004ececc382dd7c954       at java.util.zip.ZipFile.<init>(ZipFile.java:127)
      @400000004ececc382dd7cd3c       at java.util.jar.JarFile.<init>(JarFile.java:135)
      @400000004ececc382dd7cd3c       at java.util.jar.JarFile.<init>(JarFile.java:114)
      @400000004ececc382dd7cd3c       at org.apache.felix.framework.util.JarFileX.<init>(JarFileX.java:41)
      @400000004ececc382dd7d124       at org.apache.felix.framework.util.SecureAction.openJAR(SecureAction.java:611)
      @400000004ececc382dd7d124       at org.apache.felix.framework.cache.JarContent.openJarFile(JarContent.java:474)
      @400000004ececc382dd7d8f4       at org.apache.felix.framework.cache.JarContent.<init>(JarContent.java:61)
      @400000004ececc382dd7d8f4       at org.apache.felix.framework.cache.JarContent.getEntryAsContent(JarContent.java:264)
      @400000004ececc382dd7dcdc       at org.apache.felix.framework.ModuleImpl.calculateContentPath(ModuleImpl.java:584)
      @400000004ececc382dd7dcdc       at org.apache.felix.framework.ModuleImpl.initializeContentPath(ModuleImpl.java:535)
      @400000004ececc382dd7e4ac       at org.apache.felix.framework.ModuleImpl.getContentPath(ModuleImpl.java:522)
      @400000004ececc382dd7e4ac       at org.apache.felix.framework.ModuleImpl.access$600(ModuleImpl.java:73)
      @400000004ececc382dd7e894       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.findClass(ModuleImpl.java:1732)
      @400000004ececc382dd7e894       at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:716)
      @400000004ececc382dd7f064       at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73)
      @400000004ececc382dd7f44c       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690)
      @400000004ececc382dd7f44c       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      @400000004ececc382dd7f834       at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java)
      @400000004ececc382dd7fc1c       at org.apache.felix.framework.ModuleImpl.getClassByDelegation(ModuleImpl.java:634)
      @400000004ececc382dd80004       at org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:541)
      @400000004ececc382dd80004       at org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:278)
      @400000004ececc382dd803ec       at org.apache.felix.framework.Felix.getServiceReferences(Felix.java:2920)
      @400000004ececc382dd85dc4       at org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:2970)
      @400000004ececc382dd85dc4       at org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:309)
      @400000004ececc382dd861ac       at org.osgi.util.tracker.ServiceTracker.getInitialReferences(ServiceTracker.java:362)
      @400000004ececc382dd861ac       at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:311)
      @400000004ececc382dd8714c       at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:273)
      @400000004ececc382dd8714c       at com.atlassian.plugin.osgi.factory.OsgiPluginInstalledHelper.onEnable(OsgiPluginInstalledHelper.java:92)
      @400000004ececc382dd87534       at com.atlassian.plugin.osgi.factory.OsgiPlugin.enableInternal(OsgiPlugin.java:404)
      @400000004ececc382dd87534       at com.atlassian.plugin.impl.AbstractPlugin.enable(AbstractPlugin.java:237)
      @400000004ececc382dd880ec       at com.atlassian.plugin.impl.AbstractDelegatingPlugin.enable(AbstractDelegatingPlugin.java:213)
      @400000004ececc382dd884d4       at com.atlassian.plugin.manager.PluginEnabler.actualEnable(PluginEnabler.java:114)
      @400000004ececc382dd888bc       at com.atlassian.plugin.manager.PluginEnabler.enable(PluginEnabler.java:99)
      @400000004ececc382dd888bc       at com.atlassian.plugin.manager.DefaultPluginManager.addPlugins(DefaultPluginManager.java:682)
      @400000004ececc382dd88ca4       at com.atlassian.plugin.manager.DefaultPluginManager.init(DefaultPluginManager.java:156)
      @400000004ececc382dd8908c       at com.atlassian.jira.plugin.JiraPluginManager.start(JiraPluginManager.java:61)
      @400000004ececc382dd89474       at com.atlassian.jira.ComponentManager$PluginSystem.start(ComponentManager.java:676)
      @400000004ececc382dd89474       at com.atlassian.jira.ComponentManager.startPluginSystem(ComponentManager.java:242)
      @400000004ececc382dd89c44       at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:235)
      @400000004ececc382dd8a02c       at com.atlassian.jira.ComponentManager.start(ComponentManager.java:220)
      @400000004ececc382dd8a02c       at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:23)
      @400000004ececc382dd8a414       at com.atlassian.jira.startup.DefaultJiraLauncher$3.run(DefaultJiraLauncher.java:101)
      @400000004ececc382dd8a7fc       at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:250)
      @400000004ececc382dd8abe4       at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:149)
      @400000004ececc382dd8afcc       at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:94)
      @400000004ececc382dd8b3b4       at com.atlassian.jira.startup.DefaultJiraLauncher.access$100(DefaultJiraLauncher.java:24)
      @400000004ececc382dd8b79c       at com.atlassian.jira.startup.DefaultJiraLauncher$1.run(DefaultJiraLauncher.java:61)
      @400000004ececc382dd8b79c       at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:33)
      @400000004ececc382dd8bb84       at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:56)
      @400000004ececc382dd8c354       at com.atlassian.jira.startup.LauncherContextListener$1.create(LauncherContextListener.java:68)
      @400000004ececc382dd8c354       at com.atlassian.jira.startup.LauncherContextListener$1.create(LauncherContextListener.java:63)
      @400000004ececc382dd8c73c       at com.atlassian.multitenant.impl.MultiTenantComponentMapImpl.get(MultiTenantComponentMapImpl.java:121)
      @400000004ececc382dd8cb24       at com.atlassian.multitenant.impl.MultiTenantComponentMapImpl.onTenantStart(MultiTenantComponentMapImpl.java:165)
      @400000004ececc382dd8cf0c       at com.atlassian.multitenant.impl.DefaultMultiTenantManager$1.consume(DefaultMultiTenantManager.java:134)
      @400000004ececc382dd8d2f4       at com.atlassian.multitenant.impl.DefaultMultiTenantManager$1.consume(DefaultMultiTenantManager.java:131)
      @400000004ececc382dd8d2f4       at com.atlassian.multitenant.impl.DefaultMultiTenantManager.runForEachListener(DefaultMultiTenantManager.java:256)
      @400000004ececc382dd8dac4       at com.atlassian.multitenant.impl.DefaultMultiTenantManager.startTenant(DefaultMultiTenantManager.java:130)
      @400000004ececc382dd8dac4       at com.atlassian.multitenant.impl.DefaultMultiTenantManager.startAll(DefaultMultiTenantManager.java:203)
      @400000004ececc382dd8e67c       at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:96)
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            rsmart metapoint
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: