Depositor configuration¶
Model¶
To start the application, include it in model and click on Apply button.
define:
type: rc/model
services:
lmio-depositor:
- <node> # Replace with name of the node
Configuration sample¶
This is the most basic configuration required for LogMan.io Depositor:
[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
[library]
providers=zk:///library
[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092
[elasticsearch]
url=http://es01:9200
Zookeeper¶
Specify locations of the Zookeeper server in the cluster.
[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
Hint
For non-production deployments, the use of a single Zookeeper server is possible.
Library¶
Specify the path(s) to the library to load declarations from.
[library]
providers=zk:///library
Hint
Since ECS.yaml schema in /Schemas is utilized by default, consider using the LogMan.io Common Library.
Kafka¶
Specify bootstrap servers of the Kafka cluster.
[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092
Hint
For non-production deployments, the use of a single Kafka server is possible.
Elasticsearch¶
Specify URLs of Elasticsearch master nodes.
The ESConnection section is used for setting advanced parameters of the connection (see below).
The elasticsearch section is used for storing URLs and authorization items.
The asab:storage section is used to explicitly allow the storage initialization.
[asab:storage]
type=elasticsearch
[connection:ESConnection]
precise_error_handling=true
bulk_out_max_size=1582912
output_queue_max_size=5
loader_per_url=1
cluster_status_throttle=yellow
cluster_status_unthrottle=green
active_shards_percent_throttle=50
retry_errors=unavailable_shards_exception
throttle_errors=circuit_breaking_exception
[elasticsearch]
url=http://es01:9201
username=MYUSERNAME
password=MYPASSWORD
Hint
The URL should point to the Elasticsearch hot node on the same server the Depositor is deployed to.
Elasticsearch Connection Advanced Settings¶
precise_error_handling¶
Specifies Elasticsearch should return information about which events caused the issue together with the error.
bulk_out_max_size¶
Size of a single bulk being sent to Elasticsearch in bytes.
Events are grouped into bulks to lower the number of requests sent to Elasticsearch.
output_queue_max_size¶
Maximum queue size for Elasticsearch bulks.
If the number is exceeded, the given pipeline is throttled.
loader_per_url¶
Number of tasks/loaders per URL. It specifies the number of requests which can be sent simultaneously to every URL specified in the URL attribute.
cluster_status_throttle¶
The state the cluster must enter in order to stop/throttle the Depositor. Can be set to none.
Default: yellow
Options: red, yellow, none
cluster_status_unthrottle¶
The state the cluster must enter in order to resume/unthrottle the Depositor, if throttled.
Default: green
Options: red, yellow, green
active_shards_percent_throttle¶
The minimum percentage of total shards that must be active/available in order for Depositor to send events.
The value, which is a percentage, should be set to 100 / (number of replicas + 1).
Default: 50
hot_node_tag¶
Substring matched in _cat/shards node names when counting non STARTED hot tier shards for unassigned_shards_tolerance.
Default: hot
Set to none (or empty) to skip the hot tier shard check entirely. In emergency multi site outages this is simpler than guessing a large unassigned_shards_tolerance. See Elasticsearch Cluster Emergency Settings.
unassigned_shards_tolerance¶
Maximum number of non STARTED shards on nodes whose name contains hot_node_tag before Depositor treats the hot tier as not ready and throttles ingestion.
Default: 32
During a multi site outage many shards on the failed location stay non STARTED. You can raise this value (for example 100000) while keeping hot_node_tag: hot, or set hot_node_tag: none and skip the check. Has no effect when hot_node_tag is none or empty.
cluster_reroute_retry_failed¶
When cluster health is red and unassigned_shards is greater than zero, Depositor periodically calls POST /_cluster/reroute?retry_failed=true on a reachable hot URL. That clears exhausted allocation retry counters so stuck primary shards (for example after a failed ILM shrink / shard lock) can be reassigned without a manual reroute.
Default: yes
Set to no to disable. Requires Elasticsearch credentials that can manage cluster reroute. The call is skipped when health itself is unreachable (no master).
cluster_reroute_retry_failed_interval¶
Minimum seconds between retry_failed reroute attempts. Aligns with the index health check cadence (1800 = 30 minutes).
Default: 1800
retry_errors¶
List of per-document Elasticsearch error types (separated by ;) that cause the event to be retried on the same pipeline instead of being redirected to others.
Default: unavailable_shards_exception;cluster_block_exception;node_not_connected_exception
Note
This configuration option is unnecessary in most cases, so it is recommended to exclude it from your configuration file.
retry_http_status¶
List of HTTP status codes for a full bulk failure (separated by ;) that cause events to be retried on the same pipeline instead of being redirected to others.
Use this for temporary cluster / gateway outages. Document-level problems (mapping, bad payload) should still go to others.
Default: 500;502;503
Example: HTTP 503 with cluster_block_exception / no master during maintenance or datacenter disconnect must not flood the others index.
throttle_errors¶
List of per-document Elasticsearch error types (separated by ;) that pause (throttle) ingestion until cluster health recovers.
Default: circuit_breaking_exception
Note
This configuration option is unnecessary in most cases, so it is recommended to exclude it from your configuration file.
throttle_http_status¶
List of HTTP status codes for a full bulk failure (separated by ;) that also pause ingestion until cluster health recovers (same mechanism as cluster status throttle). The pause clears automatically on the next health check once throttle conditions are no longer met.
Default: 503
Set to empty (throttle_http_status= in INI, or throttle_http_status: "" in the model) to disable this pause. Failed bulks still retry when the status is listed in retry_http_status. See Elasticsearch Cluster Emergency Settings.
Declarations¶
Optional section to specify where to load the declarations of event lanes from and which schema is going to be used by default (if it is not specified in the given event lane declaration).
[declarations]
path=/EventLanes/
schema=/Schemas/ECS.yaml
Hint
Make sure to change the schema if you are using a schema other than ECS in your deployment as your default. Changing the path for event lanes is discouraged.