Event Lanes¶
TeskaLabs LogMan.io Depositor reads all event lanes from the library and creates Kafka-to-Elasticsearch pipelines based on kafka and elasticsearch sections.
Note
All deployed instances of TeskaLabs LogMan.io Depositor share the same Group ID within Kafka. All instances distribute consumption from Kafka partitions among themselves and thus provide scalability natively.
Declaration¶
This is an example of event lane sections relevant for LogMan.io Depositor:
---
define:
type: lmio/event-lane
kafka:
events:
topic: events.<tenant>.<stream> # Kafka topic for parsed events
others:
topic: others.<tenant>.<stream> # Kafka topic for unparsed events
elasticsearch:
events:
index: lmio-<tenant>-events-<stream> # Index alias for events
others:
index: lmio-<tenant>-others # Index alias for others
When LogMan.io Depositor is started and the event lane is loaded, two pipelines are created.
Events pipeline consumes messages from Kafka topic defined in kafka/events/topic option and stores them in Elasticsearch index, using index alias defined in elasticsearch/events/index option.
Others pipeline similarly consumes messages from topic defined in kafka/others/topic and stores them in index using alias defined in elasticsearch/others/index.
Index vs. index alias
In Elasticsearch, an index is a collection of documents that share the same structure and are stored together. It is the primary unit of data storage and retrieval.
An index alias is a virtual name that can point to one or multiple indices. It allows to view and manipulate with data of the same logical stream.
In event lane, index alias is specified. LogMan.io Depositor creates indices based on that alias.
For example, when the index alias is defined as lmio-tenant-events-stream, _LogMan.io Depositor creates indices
lmio-tenant-events-stream-000001
lmio-tenant-events-stream-000002
lmio-tenant-events-stream-000003
...
Complex event lanes
LogMan.io Depositor does not natively read from the events.<tenant>.complex Kafka topic and skips complex event lanes.
Note
Depositor considers ALL event lane files regardless of if they are disabled for the given tenant in the UI or not. Depositor is not a tenant-specific service.
Index template¶
LogMan.io Depositor creates and updates index template of each event lane.
The mappings in the index template are based on the event lane schema. Default schema for event lane is /Schemas/ECS.yaml. It can be changed in event lane declaration:
---
define:
type: lmio/event-lane
schema: /Schemas/CEF.yaml
It is also possible to specify number_of_shards and number_of_replicas in the settings section in elasticsearch:
---
define:
type: lmio/event-lane
elasticsearch:
events:
settings:
number_of_shards: 6
number_of_replicas: 1
The default number_of_shards is 6 and number_of_replicas is 1.
Note
Please consider carefully before changing the default settings and schema. Changing the defaults usually causes issues such as non-matching detection rules for the given event lane that uses a different schema.
Warning
Changes to the index template will only take effect after the next index rollover if an index already exists in Elasticsearch.
Lifecycle Policy¶
LogMan.io Depositor configures Index Lifecycle Policy of each event lane.
Default¶
The default lifecycle policy contains four phases: hot, warm, cold, and delete.
-
The default hot phase for the given index ends when primary shard size exceeds 16 GB or is older than 7 days.
-
The default warm phase for the given index starts either when hot ends, or after 7 days, and turns on shrinking.
-
The default cold phase for the given index starts after 14 days.
-
The delete phase deletes the index after 180 days.
Custom¶
The default ILM can be changed, even though it is not recommended for most cases. You can do so by specifying the lifecycle section within the event lane's elasticsearch section:
---
define:
type: lmio/event-lane
elasticsearch:
events:
lifecycle:
hot:
min_age: "0ms"
actions:
rollover:
max_primary_shard_size: "25gb" # We want bigger primary shards than default
max_age: "30d"
set_priority:
priority: 100
warm:
min_age: "7d"
actions:
shrink:
number_of_shards: 1
set_priority:
priority: 50
cold:
min_age: "14d"
actions:
set_priority:
priority: 0
delete:
min_age: 180d
actions:
delete:
delete_searchable_snapshot: true
Set complete ILM policy.
Even if you aim to change just one of all the phases, you need to specify whole lifecycle policy. Custom ILM overrides the default configuration completely.
No delete phase
If you don't want to setup delete phase, just omit the section delete in event lane. Use this only if you really know what you are doing!