Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-10047

Elasticsearch health checks run multiple times

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Low
    • 5.0.0, 4.14.6
    • 4.6.0
    • Search
    • None

    Description

      Under certain conditions, the search plugin can create duplicate scheduled jobs for its Elasticsearch health checks, which can result in those checks running many times. For example:

      2017-08-07 04:11:27,717 INFO  [Caesium-1-1]  c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks
      2017-08-07 04:11:27,766 INFO  [I/O dispatcher 981]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4
      2017-08-07 04:11:27,798 INFO  [Caesium-1-1]  c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H
      2017-08-07 04:20:09,768 INFO  [Caesium-1-1]  c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks
      2017-08-07 04:20:09,788 INFO  [I/O dispatcher 991]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4
      2017-08-07 04:20:09,814 INFO  [Caesium-1-1]  c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H
      2017-08-07 04:31:13,753 ERROR [AtlassianEvent::thread-6]  c.a.s.p.m.i.MergeRequestMergeCheck Issue encountered in the "check for veto" of Unbreakable Build. Exception is:java.lang.NullPointerException
      2017-08-07 04:35:36,052 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks
      2017-08-07 04:35:36,092 INFO  [I/O dispatcher 1001]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4
      2017-08-07 04:35:36,135 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H
      2017-08-07 04:42:10,161 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks
      2017-08-07 04:42:10,175 INFO  [I/O dispatcher 1011]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4
      2017-08-07 04:42:10,196 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H
      2017-08-07 04:43:09,089 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks
      2017-08-07 04:43:09,129 INFO  [I/O dispatcher 1021]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4
      2017-08-07 04:43:09,152 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H
      2017-08-07 04:55:57,195 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks
      2017-08-07 04:55:57,215 INFO  [I/O dispatcher 1031]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4
      2017-08-07 04:55:57,241 INFO  [Caesium-1-2]  c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H
      

      The messages indicate the checks are running 12 hours (PT12H) apart, but the timestamps show they're actually happening every ~10 minutes or so.

      Attachments

        Activity

          People

            khughes@atlassian.com Kristy
            bturner Bryan Turner (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: