Data Lifecycle
The data lifecycle within TeskaLabs LogMan.io encompasses multiple stages designed to manage log data efficiently across its journey. From initial ingestion, archive and parsing, indexing, and eventual deletion, each stage is carefully configured to optimize performance, data retention, and compliance needs. While our default configurations provide sensible settings for most use cases, allowing smooth operation from day one, understanding the lifecycle stages can help refine and tailor the system as your requirements evolve. This chapter offers an overview of these stages and the databases involved, laying the groundwork for advanced data management.
Data (e.g. logs, events, metrics) are stored in several availability stages, basically in the chronological order.
There are several stops in the data flow where life cycle or retention of the data can be adapted:
- Archive - The first component that collects data is LMIO Receiver which provides Archive. Data in the Archive are managed through life cycle policy set for each stream.
Data lifecycle management in Archive
- Kafka - LMIO Receiver sends data to
received.<tenant>.<stream>topic. Once parsed, data reachevents.<tenant>.<stream>topic orothers.<tenant>if parsing fails.
Rentention policy settings in Kafka
- Elasticsearch - Parsed data are stored and indexed in Elasticsearch with its data lifecycle management.
Index lifecycle management in Elasticsearch
To remind the data flow through TeskaLabs LogMan.io, please see the Architecture diagram.