Skip to content

Event Lanes

Relation to LogMan.io Parsec

TeskaLabs LogMan.io Parsec reads important part of its configuration from event lane. This configuration covers:

  • Kafka topics from which events are taken and to which topics parsed and error events are sent
  • parsing rules (declarations)
  • (optionally) timezone, charset and schema
  • group.id for consuming from received topic

Therefore, each instance of LogMan.io Parsec runs under exactly one event lane (under exactly one tenant).

Declaration

This is the minimal required event lane definition, located in the /EventLanes/<tenant> directory in the Library:

/EventLanes/mytenant/eventlane.yaml
---
define:
    type: lmio/event-lane

parsec:
    name: /Parsers/path/to/parser/  # (1)

kafka:
    received:
        topic: received.tenant.stream
    events:
        topic: events.tenant.stream
    others:
        topic: others.tenant
  1. Path for the parsing rule. It must start with /Parsers. The standard path format is <vendor>/<type>, e.g. Microsoft/IIS or Oracle/Listener, but in case only one technology is used, only the name of the provider can be used, e.g. Zabbix or Devolutions.

When Parsec is started and the event lane is loaded, two pipelines are created:

  • ParsecPipeline between received and events topic
  • ErrorPipeline targeting to others topic

group.id used for consuming from received topic has the form: lmio-parsec-<tenant>-<eventlane>

Timezone, schema, charset

Timezone, schema and charset are read from the tenant configuration by default, but these properties can be overwritten in event lane:

/EventLanes/tenant/eventlane.yaml
---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml
    timezone: UTC
    charset: utf-16

schema: What schema is used for parsing. Mappings and enrichers are schema-specific and schema must be set in declaration. If other schema is used, these declarations are omitted.

timezone: If the log source produces logs in the specific timezone, different from the tenant default timezone, it has to be specified here. The name of the timezone must be compliant with IANA Time Zone Database. Internally, all timestamps are converted into UTC.

charset: If the log source produces logs in the charset (or encoding) different from UTF-8, the charset must be specified here. The list of supported charset is here. Internally, every text is encoded in UTF-8.

Parsing options

Additional parsing options are specified also in event lane.

/EventLanes/tenant/eventlane.yaml
parsec:
    name: /Parsers/path/to/parser/
    event:
        extract:
            ip_addresses: true  # Regex extraction of IPv4 and IPv6 addresses
        drop:
            empty: true  # Empty events are not processed