Issue Summary

Insight checks for missing default attributes and creates them during Jira startup, causing Jira to take longer time to complete starting up and be accessible

This is reproducible on Data Center: yes

Steps to Reproduce

This issue may not be reproduced by will but objects without default attributes could be created with the steps below.

1. While Jira is running, delete the object attribute value and at least one of the default object attribute of some objects from the database. SQL queries below can be used.

   --Find 10 random objects and delete their Key, Updated and Created attribute values
   delete from "AO_8542F1_IFJ_OBJ_ATTR_VAL" val where val."OBJECT_ATTRIBUTE_ID" in 
   (select val."OBJECT_ATTRIBUTE_ID" from "AO_8542F1_IFJ_OBJ_ATTR_VAL" val 
   left join "AO_8542F1_IFJ_OBJ_ATTR" obj_attr on obj_attr."ID" = val."OBJECT_ATTRIBUTE_ID" 
   right join "AO_8542F1_IFJ_OBJ_TYPE_ATTR" type_attr on type_attr."ID" = obj_attr."OBJECT_TYPE_ATTRIBUTE_ID"
   where "OBJECT_ATTRIBUTE_ID" in (select obj_attr."ID" from "AO_8542F1_IFJ_OBJ_ATTR" obj_attr where obj_attr."OBJECT_ID" in (select obj."ID" from "AO_8542F1_IFJ_OBJ" obj order by random() limit 10))
   and type_attr."NAME" in ('Updated','Created','Key'));
   
   --Delete the object attributes that no longer have values (deleted with the query above)
   delete from "AO_8542F1_IFJ_OBJ_ATTR" attr where attr."ID" in (select attr."ID" 
   from "AO_8542F1_IFJ_OBJ_ATTR" attr
   left join "AO_8542F1_IFJ_OBJ_ATTR_VAL" val on val."OBJECT_ATTRIBUTE_ID" = attr."ID"
   where val."ID" IS null)
   
   --List the objects that have missing default attributes
   select O."ID" as "object ID", S."NAME" as "object schema name"
   from "AO_8542F1_IFJ_OBJ" O 
   left join "AO_8542F1_IFJ_OBJ_TYPE" OT on O."OBJECT_TYPE_ID" = OT."ID"
   right join "AO_8542F1_IFJ_OBJ_SCHEMA" S on OT."OBJECT_SCHEMA_ID" = S."ID" 
   where O."ID" IN (SELECT
     O."ID"
     FROM "AO_8542F1_IFJ_OBJ" O
     LEFT OUTER JOIN "AO_8542F1_IFJ_OBJ_ATTR" OA ON O."ID" = OA."OBJECT_ID"
   WHERE OA."OBJECT_TYPE_ATTRIBUTE_ID" IN (
     SELECT OTA."ID"
     FROM "AO_8542F1_IFJ_OBJ_TYPE" OT
       LEFT OUTER JOIN "AO_8542F1_IFJ_OBJ_TYPE_ATTR" OTA ON OTA."OBJECT_TYPE_ID" = OT."ID"
     WHERE (OTA."NAME" IN ('Key', 'Updated', 'Created') OR OTA."LABEL"))
   GROUP BY O."ID"
   HAVING count(*) < 4) 
   

2. Run an Insight reindex.
3. Go to Insight schema search and search for objects without Key, Created or Updated. The 3rd query above listed the objects with their respective schema that can be used for this search. This IQL can be used to search for corrupted objects within the given schema.

   Updated is empty or Created is empty or Key is empty
   

4. View one or more of the objects in the GUI and notice that they are all missing Key, Created and Updated attributes and values.
   • Example of an object without Updated and Created attributes which are default.
     • 
   • Example of an object with all 4 default attributes.
     • 
5. Restart Jira.
6. Check Jira log.

Expected Results

Jira's startup time (Jira is accessible) is almost the same as its last few startups. If Jira used to take about 7 minutes to start up and become accessible, it will still take about 7 minutes.

Actual Results

Jira takes longer time to be accessible. If Jira used to take about 7 minutes to start up and become accessible, it might take 10 minutes or more to become accessible.

While Jira is inaccessible, repeated entries like the following can be seen in the atlassian-jira.log.

...
2022-09-26 19:57:58,063+0200 localhost-startStop-1 WARN      [c.r.j.p.i.services.launcher.InsightLauncher] Insight is about to deploy and start.
2022-09-26 19:57:58,063+0200 localhost-startStop-1 WARN      [c.r.j.p.i.services.launcher.InsightLauncher] Control Insight for data consistancy start
2022-09-26 19:57:59,391+0200 localhost-startStop-1 WARN      [c.r.j.p.i.services.launcher.InsightLauncher] Default attributes for Created not found, creating them for object AO_8542F1_IFJ_OBJ {ID = 186336} (object 1)
2022-09-26 19:57:59,531+0200 localhost-startStop-1 WARN      [c.r.j.p.i.services.launcher.InsightLauncher] Default attributes for Created not found, creating them for object AO_8542F1_IFJ_OBJ {ID = 202275} (object 2)
2022-09-26 19:57:59,665+0200 localhost-startStop-1 WARN      [c.r.j.p.i.services.launcher.InsightLauncher] Default attributes for Created not found, creating them for object AO_8542F1_IFJ_OBJ {ID = 188855} (object 3)
2022-09-26 19:57:59,734+0200 localhost-startStop-1 WARN      [c.r.j.p.i.services.launcher.InsightLauncher] Default attributes for Created not found, creating them for object AO_8542F1_IFJ_OBJ {ID = 176668} (object 4)
...

Jira will only complete the startup and be accessible after creating all default attributes for all affected objects. Having said that, Jira startup could be delayed depending on how many objects without all 4 default attributes (Key, Label, Updated, Created) present in the instance.

This operation cannot be cancelled or disabled. If the startup is interrupted (e.g. Jira is restarted again), it will continue to create the missing default attributes as long as there are still objects without all 4 default attributes. However, if this operation can be completed, it will not run again in the next startups unless there are new objects with missing default attributes.

Workaround

To prevent Jira from taking long time to start up during your next restart, proactively look for and delete objects without Label, Key, Updated or Created attributes. These objects are likely created due to this bug - https://jira.atlassian.com/browse/JSDSERVER-11062, which they are corrupted and have to be removed anyway.
If your instance is affected by this bug, upgrade to the fix version as soon as possible to prevent more corrupted objects from being created that way. It's recommended to check the instance for objects with missing default attributes before upgrading, as an application restart will be triggered during the upgrade process and you might experience this current issue during upgrade, which prolongs the time to complete the upgrade.

This SQL query could also be used to check for objects with any missing default attributes.

SELECT
  O."ID",
  count(*)
FROM "AO_8542F1_IFJ_OBJ" O
  LEFT OUTER JOIN "AO_8542F1_IFJ_OBJ_ATTR" OA ON O."ID" = OA."OBJECT_ID"
WHERE OA."OBJECT_TYPE_ATTRIBUTE_ID" IN (
  SELECT OTA."ID"
  FROM "AO_8542F1_IFJ_OBJ_TYPE" OT
    LEFT OUTER JOIN "AO_8542F1_IFJ_OBJ_TYPE_ATTR" OTA ON OTA."OBJECT_TYPE_ID" = OT."ID"
  WHERE (OTA."NAME" IN ('Key', 'Updated', 'Created') OR OTA."LABEL"))
GROUP BY O."ID"
HAVING count(*) < 4;

Notes

Objects without default attributes cannot be detected earlier via Insight indexing, which makes this problem not easily detectable until the issue occurs