Event Lane Profile¶

Event lane profiles define the data lifecycle for event lanes. A profile specifies how long data is stored in Archive, Kafka, and Elasticsearch in each phase, and when it is deleted.

Event lane profile assignment¶

When creating or modifying an event lane, you can assign a profile to it in the event lane declaration file located in the Library at /EventLanes/tenant/eventlane.yaml.

/EventLanes/tenant/eventlane.yaml

---
define:
  type: lmio/event-lane
  profile: /Profiles/EventLanes/Profile.yaml  # Assign the profile here

By default, if no profile is specified, the /Profiles/EventLanes/Default.yaml profile is used.

Profile definition¶

Profiles are defined in the Library under /Profiles/EventLanes/. Each profile is a YAML file that specifies the lifecycle policies for Archive streams, Kafka topics, and Elasticsearch indices associated with event lanes.

Every section can be overridden in the event lane declaration, allowing for customization of lifecycle policies on a per-event-lane basis.

The following sections can be defined in a profile:

archive: Defines the lifecycle policies for Archive streams, including phases like hot, warm, and cold, and actions such as copy, move, compress, and delete. Read more about Archive lifecycle management.
kafka: Specifies the retention settings for Kafka topics, including received and events.
elasticsearch: Outlines the lifecycle policies for Elasticsearch indices, including phases like hot, warm, cold, and delete, along with actions such as rollover, shrink, set_priority, and delete. Read more about Elasticsearch lifecycle management.

Warning

Each section must be fully defined in the profile. Partial definitions are not supported. For example, if you define the elasticsearch section in a profile, you must include all necessary phases and actions within that section.

/Profiles/EventLanes/Profile.yaml

---
define:
  type: lmio/event-lane-profile
  name: Default

archive:
  lifecycle:
    hot:
    - copy:
        phase: cold
    - move:
        age: 1d
        phase: warm
    warm:
    - delete:
        age: 3M
    cold:
    - compress:
        preset: 6
        threads: 4
        type: xz
    - delete:
        age: 18M

kafka:
  received:
    settings:
      retention: 1d
  events:
    settings:
      retention: 1d

elasticsearch:
  events:
    lifecycle:
      hot:
        min_age: 0ms
        actions:
          rollover:
            max_primary_shard_size: 16gb
            max_age: 14d
          set_priority:
            priority: 100
      warm:
        min_age: 3d
        actions:
          shrink:
            number_of_shards: 1
          set_priority:
            priority: 50
      cold:
        min_age: 14d
        actions:
          set_priority:
            priority: 0
      delete:
        min_age: 180d
        actions:
          delete:
            delete_searchable_snapshot: true