Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-25484

labels upgrade task 552 is sooooooo slow

    XMLWordPrintable

Details

    • We collect Jira feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

    Description

      NOTE: This suggestion is for JIRA Server. Using JIRA Cloud? See the corresponding suggestion.

      I'm running a test upgrade from Jira 4.1.2 to 4.3.4 in preparation of a production upgrade.
      I noticed that migrating label data was taking forever.
      Looking at the code I see that UpgradeTask_Build552 does a database insert for every single label.
      Ahhhh!
      That's why it's taking sooooooo long (hours and hours).
      We don't have hours and hours and hours available when it comes time to upgrade Jira.
      We have probably half a million labels.

      I came up with an improved bit of code that collects labels in batches of 5000. It's much faster. It runs in minutes versus hours.

      
          void migrateCustomFieldData(final List<Long> fieldsToConvertToSystem, final List<Long> customFieldIds)
          {
              //Copy the data across for all custom fields!
              OfBizListIterator iterator = null;
              try
              {
                  iterator = ofBizDelegator.findListIteratorByCondition(CF_VALUE_ENTITY, new EntityExpr("customfield", EntityOperator.IN,
                          customFieldIds), null, EasyList.build("issue", "customfield", VALUE), null, null);
                  final List<GenericValue> labelRows = new ArrayList<GenericValue>(BATCH);
                  int i = 0;
      
                  for (GenericValue gv = iterator.next(); gv != null; gv = iterator.next())
                  {
                      final Long issueId = gv.getLong("issue");
                      final Long customFieldId = gv.getLong("customfield");
                      final Collection<String> labels = LabelParser.buildFromString(new PassStringThrough(), gv.getString(VALUE));
                      if (labels != null)
                      {
                          final Map<String, Object> params = MapBuilder.<String, Object>newBuilder().add(OfBizLabelStore.Columns.ISSUE_ID, issueId).toMutableMap();
                          if (!fieldsToConvertToSystem.contains(customFieldId))
                          {
                              params.put(OfBizLabelStore.Columns.CUSTOM_FIELD_ID, customFieldId);
                          }
                          for (String label : labels)
                          {
                              params.put(OfBizLabelStore.Columns.LABEL, label);
                              labelRows.add(createValue(OfBizLabelStore.TABLE, params));
                              if(++i == BATCH) {
                                 ofBizDelegator.storeAll(labelRows);
                                 labelRows.clear();
                                 i = 0;
                              }
                          }
                      }
                  }
                  ofBizDelegator.storeAll(labelRows);
              }
              catch(GenericEntityException gee) {
                 throw new DataAccessException(gee);
              }
              finally
              {
                  if (iterator != null)
                  {
                      iterator.close();
                  }
              }
          }
      
         /**
          * Create a new entity.
          *
          * If there is no "id" in the parameter list, one is created using the entity sequence
          */
         // From EntityUtils
         private static GenericValue createValue(final String entity, final Map<String, ? extends Object> paramMap) throws GenericEntityException
         {
             final Map<String, Object> params = (paramMap == null) ? new HashMap<String, Object>() : new HashMap<String, Object>(paramMap);
      
             if (params.get("id") == null)
             {
                 final Long id = CoreFactory.getGenericDelegator().getNextSeqId(entity);
                 params.put("id", id);
             }
      
             return CoreFactory.getGenericDelegator().makeValue(entity, params);
         }
      

      Attachments

        Issue Links

          Activity

            People

              pniewiadomski Pawel Niewiadomski (Inactive)
              adc6ee404f6d Jeff Kirby
              Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: