Elasticsearch Cluster Emergency Settings

When Elasticsearch is degraded, LogMan.io Depositor can pause (throttle) event ingestion until cluster health recovers. In an emergency you can relax those checks so Depositor keeps writing while the surviving nodes can still accept data.

Use this only for a temporary outage. Revert to defaults after the cluster is healthy again.

How Depositor decides to throttle

Every minute Depositor reads Elasticsearch cluster health. It always continues when status equals cluster_status_unthrottle (default green).

Otherwise it throttles if any of these is true:

1. Cluster status equals cluster_status_throttle (default yellow).
2. active_shards_percent_as_number is below active_shards_percent_throttle (default 50).
3. Too many non STARTED shards on nodes whose name contains hot_node_tag (default hot), above unassigned_shards_tolerance (default 32). This check is skipped when hot_node_tag is none or empty.

A single unreachable hot URL does not throttle the whole Depositor by itself: Depositor tries every configured url / url_hot endpoint before treating the cluster as offline.

When to use emergency settings

Typical cases:

• Cluster is yellow (replicas missing, primaries still up) and you want ingestion to continue without waiting for green.
• One site or location of a multi site Elasticsearch cluster is down, overall health is red, and the surviving hot nodes can still store events.

Risks while the cluster is red

While status is red, some primary shards may be missing. Writes to those indices fail even if Depositor is not throttled. Prefer this only when the surviving location still holds writable primaries for the streams you care about. Fix allocation and restore the failed site as soon as possible.

Writes when the primary shard is on the failed site

Elasticsearch needs an active primary shard to accept a write. If that primary sits only on the failed location (no replica to promote on the surviving site), indexing into that index fails even when Depositor is not throttled.

Typical Elasticsearch response:

• HTTP 503 (sometimes 500) with unavailable_shards_exception / UnavailableShardsException
• Or a per document bulk item error of type unavailable_shards_exception

Cluster status is red because at least one primary is unavailable. If only replicas are missing, status is yellow and writes to healthy primaries can still succeed (subject to wait_for_active_shards, default 1).

What the master does next:

• If a replica of the shard exists on a living node, the master promotes it to primary. The index moves toward yellow and writes resume.
• If there is no replica (zero replicas, or all copies were on the failed site), the index stays red until the original site returns or an administrator intervenes.

Depositor treats this as retryable

These failures are not sent to others. Depositor retries them on the same pipeline:

• Per document: unavailable_shards_exception is in default retry_errors (together with cluster_block_exception and node_not_connected_exception).
• Full bulk HTTP: default retry_http_status includes 500, 502, and 503.

By default, full bulk HTTP 503 also throttles ingestion (throttle_http_status=503). That pause clears automatically on the next cluster health check (about every 60 seconds) once health no longer meets the throttle conditions (cluster_status_unthrottle, cluster_status_throttle, active_shards_percent_throttle, hot_node_tag). To keep retrying 503 without pausing the pipeline, set throttle_http_status to an empty value (see emergency example below). Retry still uses retry_http_status.

Events stay in Kafka / the pipeline retry loop until a primary becomes available again (replica promotion or site recovery). Mapping or payload errors still go to others as usual.

See also Depositor Configuration (retry_errors, retry_http_status, throttle_http_status) and Depositor Troubleshooting.

Automatic allocation retry while red

Independently of emergency throttle settings, when cluster health is red with unassigned shards Depositor periodically calls POST /_cluster/reroute?retry_failed=true (default every 30 minutes; cluster_reroute_retry_failed=yes). That clears exhausted allocation retries so stuck primaries (for example after a failed ILM shrink) can recover without a manual reroute. Disable with cluster_reroute_retry_failed=no.

Keep writing when one location is down (cluster red)

Default Depositor stops ingestion when cluster status is yellow (and typically also while red, via active shard percent / hot tier checks). That is the usual state after one site fails, even when the other site still has hot capacity.

To keep depositing through that outage, disable status based throttling (cluster_status_throttle: none), lower the active shard percentage floor, and relax the hot tier shard check. Preferred way is hot_node_tag: none (skips that check entirely). Alternatively keep hot_node_tag: hot and raise unassigned_shards_tolerance high enough for leftovers on the failed site. Optionally clear throttle_http_status so a full bulk HTTP 503 retries without pausing the pipeline.

Configuration in the model

Edit /Site/model.yaml in the Library (see Configure ASAB services from the model), then apply the model:

define:
  type: rc/model

services:
  lmio-depositor:
    instances:
      - lma1
      - lma2
    asab:
      config:
        "connection:ESConnection":
          cluster_status_throttle: none
          cluster_status_unthrottle: yellow
          active_shards_percent_throttle: 25
          hot_node_tag: none
          throttle_http_status: ""
          # Alternative to hot_node_tag: none (keep default hot_node_tag=hot):
          # unassigned_shards_tolerance: 100000

Replace lma1 / lma2 with your Depositor instance nodes. After Apply, Depositor reloads the generated config.

Equivalent INI (for reference)

The model block above becomes this INI section in the Depositor config file:

[connection:ESConnection]
cluster_status_throttle=none
cluster_status_unthrottle=yellow
active_shards_percent_throttle=25
hot_node_tag=none
throttle_http_status=
; Alternative to hot_node_tag=none (keep default hot_node_tag=hot):
; unassigned_shards_tolerance=100000

What each option does in this scenario

Option	Suggested emergency value	Why
cluster_status_throttle	none	Do not pause only because overall status is red.
cluster_status_unthrottle	yellow	Treat yellow as recovered enough to clear a previous throttle without waiting for full green.
active_shards_percent_throttle	25 (tune to your topology)	One lost site often drops active shard percent near or below the default 50.
hot_node_tag	none	Preferred: skip the hot tier _cat/shards check entirely. Then you do not need to pick a tolerance.
unassigned_shards_tolerance	large number (for example 100000)	Alternative when you keep hot_node_tag: hot: allow many non STARTED shards on the failed hot nodes instead of disabling the check. Unused when hot_node_tag is none.
throttle_http_status	empty ("" / throttle_http_status=)	Do not pause the pipeline on full bulk HTTP 503. Events still retry via retry_http_status. Leave default 503 if you want a pause until the next successful health check.
Tune active_shards_percent_throttle to your replica layout. A common starting point is roughly 100 / (number of replicas + 1), then lower further if the surviving site still has free capacity and you still see throttle logs.
Watch Depositor logs for Throttling event ingestion because Elasticsearch cluster health is degraded and the reasons field; they show which check still fires.

Milder case: continue on yellow

If the cluster is only yellow and you do not want to allow writes while status is red, set only the unthrottle threshold:

services:
  lmio-depositor:
    instances:
      - lma1
    asab:
      config:
        "connection:ESConnection":
          cluster_status_unthrottle: yellow

Defaults then still throttle on red, and resume when status reaches yellow or better (instead of waiting for green).

Parameter reference

cluster_status_throttle

Cluster status that causes Depositor to pause ingestion (when status is not already equal to cluster_status_unthrottle).

• Default: yellow
• Options: red, yellow, none

cluster_status_unthrottle

If current cluster status equals this value, Depositor does not throttle (other checks are skipped for that tick).

• Default: green
• Options: red, yellow, green

active_shards_percent_throttle

Minimum active_shards_percent_as_number from cluster health. Below this value Depositor throttles (when status is not cluster_status_unthrottle).

• Default: 50

hot_node_tag

Substring matched in _cat/shards node names when counting non STARTED hot tier shards.

• Default: hot
• Set to none (or leave empty) to disable this check. Then unassigned_shards_tolerance has no effect.

unassigned_shards_tolerance

How many non STARTED shards on nodes matching hot_node_tag are allowed before Depositor treats the hot tier as not ready.

• Default: 32
• Unused when hot_node_tag is none or empty.

throttle_http_status

HTTP status codes for a full bulk failure (separated by ;) that pause ingestion until the periodic cluster health check clears the throttle.

• Default: 503
• Set to empty to disable pause on those codes. Retry of the same statuses still follows retry_http_status (default 500;502;503).

Reverting to defaults

When Elasticsearch is healthy again (green), remove the emergency overrides from lmio-depositor.asab.config in the model and apply again so Depositor returns to defaults (cluster_status_throttle=yellow, cluster_status_unthrottle=green, active_shards_percent_throttle=50, hot_node_tag=hot, throttle_http_status=503).

Related documentation

• Configure ASAB services from the model
• Depositor Configuration
• Depositor Troubleshooting