Depositor configuration¶

Model¶

To start the application, include it in model and click on Apply button.

/Site/model.yaml

define:
  type: rc/model

services:
  lmio-depositor:
    - <node>  # Replace with name of the node

Configuration sample¶

This is the most basic configuration required for LogMan.io Depositor:

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

[library]
providers=zk:///library

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

[elasticsearch]
url=http://es01:9200

Zookeeper¶

Specify locations of the Zookeeper server in the cluster.

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Hint

For non-production deployments, the use of a single Zookeeper server is possible.

Library¶

Specify the path(s) to the library to load declarations from.

[library]
providers=zk:///library

Hint

Since ECS.yaml schema in /Schemas is utilized by default, consider using the LogMan.io Common Library.

Kafka¶

Specify bootstrap servers of the Kafka cluster.

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

Hint

For non-production deployments, the use of a single Kafka server is possible.

Elasticsearch¶

Specify URLs of Elasticsearch master nodes.

The ESConnection section is used for setting advanced parameters of the connection (see below).

The elasticsearch section is used for storing URLs and authorization items.

The asab:storage section is used to explicitly allow the storage initialization.

[asab:storage]
type=elasticsearch

[connection:ESConnection]
precise_error_handling=true
bulk_out_max_size=1582912
output_queue_max_size=5
loader_per_url=1
cluster_status_throttle=red
cluster_status_unthrottle=green
active_shards_percent_throttle=50
retry_errors=unavailable_shards_exception
throttle_errors=circuit_breaking_exception

[elasticsearch]
url=http://es01:9201
username=MYUSERNAME
password=MYPASSWORD

Hint

The URL should point to the Elasticsearch hot node on the same server the Depositor is deployed to.

Elasticsearch Connection Advanced Settings¶

precise_error_handling¶

Specifies Elasticsearch should return information about which events caused the issue together with the error.

bulk_out_max_size¶

Size of a single bulk being sent to Elasticsearch in bytes.

Events are grouped into bulks to lower the number of requests sent to Elasticsearch.

output_queue_max_size¶

Maximum queue size for Elasticsearch bulks.

If the number is exceeded, the given pipeline is throttled.

loader_per_url¶

Number of tasks/loaders per URL. It specifies the number of requests which can be sent simultaneously to every URL specified in the URL attribute.

cluster_status_throttle¶

The state the cluster must enter in order to stop/throttle the Depositor. Can be set to none.

Default: red

Options: red, yellow, none

cluster_status_unthrottle¶

The state the cluster must enter in order to resume/unthrottle the Depositor, if throttled.

Default: green

Options: red, yellow, green

active_shards_percent_throttle¶

The minimum percentage of total shards that must be active/available in order for Depositor to send events.

The value, which is a percentage, should be set to 100 / (number of replicas + 1).

Default: 50

retry_errors¶

List of per-document Elasticsearch error types (separated by ;) that cause the event to be retried on the same pipeline instead of being redirected to others.

Default: unavailable_shards_exception;cluster_block_exception;node_not_connected_exception

Note

This configuration option is unnecessary in most cases, so it is recommended to exclude it from your configuration file.

retry_http_status¶

List of HTTP status codes for a full bulk failure (separated by ;) that cause events to be retried on the same pipeline instead of being redirected to others.

Use this for temporary cluster / gateway outages. Document-level problems (mapping, bad payload) should still go to others.

Default: 500;502;503

Example: HTTP 503 with cluster_block_exception / no master during maintenance or datacenter disconnect must not flood the others index.

throttle_errors¶

List of per-document Elasticsearch error types (separated by ;) that pause (throttle) ingestion until cluster health recovers.

Default: circuit_breaking_exception

Note

This configuration option is unnecessary in most cases, so it is recommended to exclude it from your configuration file.

throttle_http_status¶

List of HTTP status codes for a full bulk failure (separated by ;) that also pause ingestion until cluster health recovers (same mechanism as cluster status throttle).

Default: 503

Declarations¶

Optional section to specify where to load the declarations of event lanes from and which schema is going to be used by default (if it is not specified in the given event lane declaration).

[declarations]
path=/EventLanes/
schema=/Schemas/ECS.yaml

Hint

Make sure to change the schema if you are using a schema other than ECS in your deployment as your default. Changing the path for event lanes is discouraged.