Ship JIRA with defaults that enable log rotation

Problem Definition

I understand that there are limitations to logging based on log4j and in Tomcat but it is a pain for an Administrator, and for Atlassian support engineers to work with massive logfiles.
After some time catalina.out can grow significantly (50+ GB) and system can run out of space. Also at Windows it's not possible to log-rotate catalina.out without stopping JAVA due to platform restriction.

Suggested Solution

• Either ship Jira with a logging system which has log rotation enabled by default for catalina.out
• Or reduce logging for catalina.out implemented, see JRASERVER-67796 for more details

Notes

• catalina.out is a file attached to STDOUT and STDERR of the JAVA process.
  • Snippet from catalina.sh

    #   CATALINA_OUT    (Optional) Full path to a file where stdout and stderr
    #                   will be redirected.
    #                   Default is $CATALINA_BASE/logs/catalina.out
    ...
    eval $_NOHUP "\"$_RUNJAVA\"" "\"$LOGGING_CONFIG\"" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS \
          -Djava.endorsed.dirs="\"$JAVA_ENDORSED_DIRS\"" -classpath "\"$CLASSPATH\"" \
          -Dcatalina.base="\"$CATALINA_BASE\"" \
          -Dcatalina.home="\"$CATALINA_HOME\"" \
          -Djava.io.tmpdir="\"$CATALINA_TMPDIR\"" \
          org.apache.catalina.startup.Bootstrap "$@" start \
          >> "$CATALINA_OUT" 2>&1 "&"
    ...
    

• at the same time, _catalina.XXXX-YY-ZZ.log _files generated and controlled by Tomcat itself. Those files are log-rotated by Tomcat.
• Jira has internal Logrotate (log4j and juli) for everything it can control as Java process. catalina.out is outside the Java process.

Workaround

Choose one them:

• Options to rotate the catalina.out and detailed steps for using logrotate are documented at How to Rotate the Catalina Log File.
  • For Windows see separate KB: How to rotate the tomcat catalina log file on windows
• Increase logging level for console loggers to WARN and restart JIRA, see diff:

  diff -u ./atlassian-jira-6.4.13-standalone//atlassian-jira/WEB-INF/classes/log4j.properties  ./log4j.properties
  --- ./atlassian-jira-6.4.13-standalone//atlassian-jira/WEB-INF/classes/log4j.properties	2016-02-25 06:10:30.000000000 +0100
  +++ ./log4j.properties 	2016-09-26 15:51:16.000000000 +0200
  @@ -28,12 +28,12 @@
   #####################################################
  
   log4j.appender.console=org.apache.log4j.ConsoleAppender
  -log4j.appender.console.Threshold=DEBUG
  +log4j.appender.console.Threshold=WARN
   log4j.appender.console.layout=org.apache.log4j.PatternLayout
   log4j.appender.console.layout.ConversionPattern=%d %t %p %X{jira.username} %X{jira.request.id} %X{jira.request.assession.id} %X{jira.request.ipaddr} %X{jira.request.url} [%c{4}] %m%n
  
   log4j.appender.nowarnconsole=org.apache.log4j.ConsoleAppender
  -log4j.appender.nowarnconsole.Threshold=DEBUG
  +log4j.appender.nowarnconsole.Threshold=WARN
   log4j.appender.nowarnconsole.layout=org.apache.log4j.PatternLayout
   log4j.appender.nowarnconsole.layout.ConversionPattern=%d %t %p %X{jira.username} %X{jira.request.id} %X{jira.request.assession.id} %X{jira.request.ipaddr} %X{jira.request.url} [%c{4}] %m%n
  

• Remove all console from log specific logging events, see examples

  -log4j.logger.com.atlassian = WARN, console, filelog
  +log4j.logger.com.atlassian = WARN,  filelog
   log4j.additivity.com.atlassian = false
  

  sed code:

  sed '230,$s/console,//' ./log4j.properties > ./log4j.properties_new