Custom Fields are missing in Jira Issue Macro after upgrade to Confluence 7.4.1, 7.5.0, 7.6.0

Issue Summary

Since upgrading to Confluence 7.4.1 and 7.5.0, customers are reporting that values of custom fields are not returning and not being displayed

Bug Behaviour

The behaviour of the bug is in such a way that if a user’s Jira Issue Macro has a custom fields to be returned from Jira, these fields are not retuned.
This issue is not observed as at first, but occurs after some time. Some customers, report that it occurs after 1-2 days.

Steps to Reproduce

This is still not clear as the issue does not present it self straight after upgrade. Currently still under investigation. 

Expected Results

Requested custom fields are returned from Jira to Confluence and displayed. 

Actual Results

Requested custom fields are not returned from Jira after some few days after upgrade, or Confluence restart. 

Diagnosis

1. Identify what are the are the Custom Fields that are missing from the Jira Issue Macro table. A screenshot from the customer would help.
2. Next, determine the request that is being sent to Jira. To do this, enable the following:
   1. On the Jira side
      1. Navigate to Jira Administration » System » Logging and profiling
         1. Enable HTTP Access Logging
         2. Enable HTTP Dump Logging
   2. On the Confluence side
      1. Navigate to Confluence Administration » General Configuration » Logging and profiling
         • Add org.apache.http to the Logging Box and set to DEBUG
         • Head over to an affected page, and refresh the page. Take note of the time (for timestamp) purposes. 
      2. Could you please create and send me the Confluence support zip. You can do so by heading over to General Administration > Troubleshooting and support tools > Create support zip. The new logs will include the org.apache.http logs we've recently added. 
      3. Remove the  org.apache.http from logging as keeping it enabled will flood the logs.
   3. Back On the Jira side
      1. Could you please create and send me the Jira support zip. You can do so by heading over to System > Troubleshooting and support tools > Create support zip. The new logs will include the inbound and outbound HTTP logs we've recently enabled.
      2. Navigate to Jira Administration » System » Logging and profiling
         1. Disable HTTP Dump Logging
         2. Disable HTTP Access Logging

What to look for

Customers who are reporting the issue seems to fall into two categories as this issue manifests differently:

First category:

1. In the Confluence logs look for the HTTP request that is being sent to Jira that includes the JQL. It will look like something like this:

   1. 2020-07-10 14:01:02,619 DEBUG [JIM Marshaller:thread-5] [org.apache.http.headers] onRequestSubmitted http-outgoing-91 >> GET /jira/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml?tempMax=20&returnMax=true&jqlQuery=key+%3D+SP-23++&field=status&field=resolution&field=customfield_10200&field=customfield_10300&field=customfield_10301&field=link HTTP/1.1

1. Part of the above record is the exact JQL that is sent to JIRA:

   1. SearchRequest.xml?tempMax=20&returnMax=true&jqlQuery=key+%3D+SP-23++&field=status&field=resolution&field=customfield_10200&field=customfield_10300&field=customfield_10301&field=link HTTP/1.1

      Notice that the last few parameter are the custom fields that are being requested. The HTTP request must include these parameters. field=customfield_10200&field=customfield_10300&field=customfield_10301

1. If the request do not include the custom field parameter when it should, then you may be affected by this issue.
   Second category:
2. In the Confluence logs look for the HTTP request that is being sent to Jira that includes the JQL. It will look like something like this:
   1.  

      2020-06-24 11:08:45,254 DEBUG [JIM Marshaller:thread-65] [org.apache.http.wire] wire http-outgoing-98257 >> "GET /sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml?tempMax=300&returnMax=true&jqlQuery=issuetype+%3D+%22Project+Information%22++++&field=summary&field=type&field=created&field=updated&field=due&field=assignee&field=reporter&field=priority&field=status&field=resolution&field=project+manager&field=organisation&field=link"

1. Part of the above record is the exact JQL that is sent to JIRA:
2.  

   SearchRequest.xml?tempMax=300&returnMax=true&jqlQuery=issuetype+%3D+%22Project+Information%22++++&field=summary&field=type&field=created&field=updated&field=due&field=assignee&field=reporter&field=priority&field=status&field=resolution&field=project+manager&field=organisation&field=link

   Notice the last few fields:

     field=project+manager&field=organisation
   

1. 1. The above fields seems to be custom fields as a quick check in Jira, these fields are not by default found there.
   2. The above GET request seems to indicate that Confluence does send the request back to Jira.
2. In the Jira dump logs, we see that Jira does receive the request and the following are the list of fields Jira recognises:

   1.  Request URL Parameters : Request URL Parameters : tempMax=300 returnMax=true jqlQuery=issuetype = "Project Information" field=summary field=type field=created field=updated field=due field=assignee field=reporter field=priority field=status field=resolution field=project manager field=organisation field=link

1. 1. Project Manager and Organistion fields are among the fields that Jira recognises above, however, they are not fetched in Jira and are not sent back to Confluence.
   2. The following are the values that are sent back to Confluence per Jira issue. Notice that the Project Manager and Organisation are not included.

   3. <item> 
      <link>https://<base-url>/browse/project-123</link> 
      <key id="22884">project-123</key> 
      <summary>Template</summary> 
      <type id="10204" iconUrl="https://<base-url>/secure/viewavatar?size=xsmall&amp;avatarId=10313&amp;avatarType=issuetype">Project Information</type> 
      <priority id="3" iconUrl="https://<base-url>/images/icons/priorities/medium.svg">Medium</priority> <status id="10001" iconUrl="https://<base-url>/images/icons/status_generic.gif" description="">Done</status> 
      <statusCategory id="3" key="done" colorName="green"/> 
      <resolution id="-1">Unresolved</resolution> 
      <assignee username="-1">Unassigned</assignee> 
      <reporter username="Faridi">Ahmad Faridi</reporter> 
      <created>Thu, 15 Mar 2018 11:04:59 +0000</created> 
      <updated>Thu, 11 Oct 2018 10:31:43 +0100</updated> 
      <due></due> 
      </item>

Looking at the two paths the issue manifests, it looks like:

• When custom fields requests are in the format:field=customfield_10200 i.e. custom field ID, Confluence omits the custom field altogether from the request.

• When custom fields requests are in the format: field=custom field name i.e. custom field name, Confluence does send the request but Jira does not return it.

Workaround

Known workarounds include:

• Disabling the Confluence Jira Macro from the System Plugins and enabling it after few minutes. (this workaround doesn’t work 100% of times, sometimes it needs to be done multiple times)
  • Note: Disabling this plugin also disables its dependencies, including confluence-jira-content and confluence-software-blueprints. These bundled plugins must be manually Enabled again after the Confluence Jira Macro is Enabled.
• Restarting Confluence.
• Downgrade confluence-jira-plugin to 11.0.14 for Confluence 7.4.1
  • Please note that the JAR file is available for download from this bug ticket.