Low Under certain conditions, the search plugin can create duplicate scheduled jobs for its Elasticsearch health checks, which can result in those checks running many times. For example:
2017-08-07 04:11:27,717 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:11:27,766 INFO [I/O dispatcher 981] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:11:27,798 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:20:09,768 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:20:09,788 INFO [I/O dispatcher 991] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:20:09,814 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:31:13,753 ERROR [AtlassianEvent::thread-6] c.a.s.p.m.i.MergeRequestMergeCheck Issue encountered in the "check for veto" of Unbreakable Build. Exception is:java.lang.NullPointerException 2017-08-07 04:35:36,052 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:35:36,092 INFO [I/O dispatcher 1001] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:35:36,135 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:42:10,161 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:42:10,175 INFO [I/O dispatcher 1011] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:42:10,196 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:43:09,089 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:43:09,129 INFO [I/O dispatcher 1021] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:43:09,152 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:55:57,195 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:55:57,215 INFO [I/O dispatcher 1031] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:55:57,241 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H
The messages indicate the checks are running 12 hours (PT12H) apart, but the timestamps show they're actually happening every ~10 minutes or so.
[BSERV-10047] Elasticsearch health checks run multiple times
| Workflow | Original: Stash Workflow - Restricted [ 2422287 ] | New: JAC Bug Workflow v3 [ 3136946 ] |
| Symptom Severity | Original: Minor [ 14432 ] | New: Severity 3 - Minor [ 15832 ] |
| Assignee | New: Kristy [ khughes@atlassian.com ] |
| Fix Version/s | New: 4.14.6 [ 72034 ] | |
| Fix Version/s | New: 5.0.0 [ 66696 ] | |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: Needs Triage [ 10030 ] | New: Closed [ 6 ] |
| Description |
New:
Under certain conditions, the search plugin can create duplicate scheduled jobs for its Elasticsearch health checks, which can result in those checks running many times. For example:
{noformat} 2017-08-07 04:11:27,717 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:11:27,766 INFO [I/O dispatcher 981] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:11:27,798 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:20:09,768 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:20:09,788 INFO [I/O dispatcher 991] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:20:09,814 INFO [Caesium-1-1] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:31:13,753 ERROR [AtlassianEvent::thread-6] c.a.s.p.m.i.MergeRequestMergeCheck Issue encountered in the "check for veto" of Unbreakable Build. Exception is:java.lang.NullPointerException 2017-08-07 04:35:36,052 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:35:36,092 INFO [I/O dispatcher 1001] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:35:36,135 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:42:10,161 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:42:10,175 INFO [I/O dispatcher 1011] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:42:10,196 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:43:09,089 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:43:09,129 INFO [I/O dispatcher 1021] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:43:09,152 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H 2017-08-07 04:55:57,195 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Running up to 2 search health checks 2017-08-07 04:55:57,215 INFO [I/O dispatcher 1031] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 2.3.4 2017-08-07 04:55:57,241 INFO [Caesium-1-2] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT12H {noformat} The messages indicate the checks are running 12 hours (PT12H) apart, but the timestamps show they're actually happening every ~10 minutes or so. |
| Security | Original: Reporter and Atlassian Staff [ 10751 ] |
| Summary | Original: Search schedules Elasticsearch health checks multiple times | New: Elasticsearch health checks run multiple times |
Spam Detector Bot
made changes -
| Security | Original: Atlassian Staff [ 10750 ] | New: Reporter and Atlassian Staff [ 10751 ] |
| Security | Original: Reporter and Atlassian Staff [ 10751 ] | New: Atlassian Staff [ 10750 ] |
Spam Detector Bot
made changes -
| Security | New: Reporter and Atlassian Staff [ 10751 ] |