Skip to content

Schema in TeskaLabs LogMan.io

TeskaLabs LogMan.io uses schemas to standardize and normalize log data from diverse sources. Schemas ensure that logs from different systems, applications, and devices are mapped to a common structure, making them easier to read, analyze, and correlate. This normalization improves readability and accessibility, allowing analysts to work with logs from various sources without needing to understand each source's unique format.

Why Use Schemas?

  • Normalization: Schemas transform heterogeneous log formats into a unified structure, enabling consistent querying and analysis.
  • Readability: Standardized fields make logs easier to interpret, reducing ambiguity and improving efficiency for analysts.
  • Accessibility: With a common schema, logs from different sources can be accessed and compared seamlessly, supporting cross-system investigations and reporting.

ECS Schema

TeskaLabs LogMan.io adopts the Elastic Common Schema (ECS) as its primary schema for log normalization. ECS defines a set of fields for structuring event data, making it easier to ingest, search, and visualize logs.

Important ECS Fields

Datetime Fields

  • @timestamp: The date and time when the event occurred, typically parsed from the event itself.
  • event.created: The date and time when the event was collected by LogMan.io Collector.
  • event.ingested: The date and time when the event was received by the LogMan.io Receiver (central system) and stored in the Archive.

The chronological order of timestamps is as follows:

@timestamp < event.created < event.ingested

Base Fields

  • event.original: The original log message, preserved for reference.
  • message: The human-readable portion of the event, if present, is typically stored in this field.
  • _id: A unique identifier for the event. In TeskaLabs LogMan.io, this refers to the row_id in the Archive.
  • related.ip: A list of all IP addresses observed in the event.

Event Fields

  • event.category: The high-level category of the event (e.g., authentication, network, file), which helps group similar events.
  • event.action: The specific action performed (e.g., user login, file access).
  • event.outcome: The result of the event. Possible values are: success, failure, or unknown.
  • event.type: Describes the type of event, such as info, error, start, end, or access. This field further classifies the nature of the event within its category.
  • event.kind: Indicates the general kind of event, such as event, alert, metric, or state. This field is useful for distinguishing between regular events, alerts, and other data types.

Identifiers

  • host.id: A unique identifier for the host where the event originated. In TeskaLabs LogMan.io, this is typically enriched from host.hostname using lookups.
  • user.id: A unique identifier for the user associated with the event. In TeskaLabs LogMan.io, this is typically enriched from user.name using lookups.

Other Common Fields

  • log.level: The original log level of the event (e.g., info, warning, error).
  • source.ip and destination.ip: The IP addresses involved in the event, which are crucial for network analysis.
  • process.name and process.pid: Information about the process involved, useful for system, endpoint, and application logs.

LogMan.io Specific Fields

  • lmio.source: Identifies the origin of the event. For events collected using syslog, this is further enriched into the following fields:
    • lmio.logsource.ip: The IP address of the device from which the log originated.
    • lmio.logsource.port: The source port of the device from which the log originated.
    • lmio.logsource.protocol: The protocol used for log forwarding (e.g., tcp, udp, tls).
  • tenant: The name of the tenant to which the event belongs.

Others Schema

Others schema is used for error events, i.e. for events that are not successfully parsed. It is derived from ECS schema.

Important Others Schema Fields

  • event.dataset: Identifies the Event Lane from which the event originated.
  • event.original: The original log.
  • @timestamp: The datetime when the event was processed through parsing.
  • error.id: A unique identifier of the error.
  • error.message: A description of the exception why the event was not parsed successfully.