Elasticsearch Cluster Emergency Settings¶

In emergency situations when the Elasticsearch cluster is degraded or experiencing issues, you may need to adjust the Depositor configuration to allow data ingestion to continue even when the cluster is not in a fully healthy state.

When to Use Emergency Settings¶

Emergency cluster settings should be used when:

The Elasticsearch cluster status is yellow (some replicas are missing, but primary shards are available)
The cluster is experiencing temporary issues but can still accept writes
You need to prevent data loss by allowing the Depositor to continue writing despite cluster health warnings
The cluster is in a degraded state but operational

Use with Caution

Emergency settings should only be used temporarily during cluster recovery. Once the cluster is fully healthy, these settings should be reverted to their defaults to ensure proper data protection and cluster health monitoring.

Configuring Depositor for Emergency Mode¶

By default, the Depositor throttles (stops writing) when the Elasticsearch cluster status is red and only resumes when the status returns to green. In emergency situations, you can configure the Depositor to resume operations when the cluster status is yellow.

Configuration in model.yaml¶

Edit the model.yaml file in the Library to configure the Depositor's emergency settings:

lmio-depositor:
  instances:
    - mynode
  asab:
    config:
      "connection:ESConnection":
        cluster_status_unthrottle: yellow

Configuration Parameters¶

cluster_status_unthrottle: The minimum Elasticsearch cluster status required for the Depositor to resume operations after being throttled.
Default: green (cluster must be fully healthy)
Emergency setting: yellow (allows writes when cluster is degraded but operational)
Options: red, yellow, green

Understanding Cluster Status

green: All primary and replica shards are active. Cluster is fully operational.
yellow: All primary shards are active, but some replicas are missing. Cluster is operational but degraded.
red: Some primary shards are missing. Cluster may have data loss and should not accept writes.

Complete Example¶

Here is a complete example of emergency Depositor configuration:

lmio-depositor:
  instances:
    - lma1
    - lma2
  asab:
    config:
      "connection:ESConnection":
        cluster_status_throttle: red      # Stop writing if cluster is red
        cluster_status_unthrottle: yellow # Resume writing when cluster is yellow or better

Reverting to Default Settings¶

Once the Elasticsearch cluster has fully recovered and is in green status, remove the configuration from the model to apply default values.

For more information about Depositor configuration options, see: