Data sources¶

Data sources are named connections to datasets within a backend. Each source is exposed to the API with a provider prefix:

Prefix	Backend	Example ID
`es.`	Elasticsearch	`es.events`
`ix.`	InfluxDB	`ix.cpu`
`db.`	DuckDB	`db.ingest`

Discovery¶

Data sources are assembled from several sources:

Library (/DataSources/)

YAML declarations with define.type of datasource/elasticsearch or datasource/influxdb and a discover section. Reloaded on Library change.

InfluxDB auto-discovery

All measurements in the configured database are exposed under the system tenant (no RBAC resources).

DuckDB (built-in)

Per-tenant in-memory sources: db.ingest (ingest rates) and db.eps (events per second). Not configured in the Library.

API properties¶

Each data source exposes:

Property	Description
`type`	`elasticsearch`, `influxdb`, or `duckdb`
`label`	Display name
`order`	Sort priority (lower = first)
`tenant`	Owning tenant
`resources`	RBAC resources required for access (not returned in API; used for filtering)

List data sources: GET /{tenant}/datasources?types=elasticsearch,influxdb

Library layout¶

Data sources are defined in the Library. Each data source definition must be on global target, not tenant.

Paths are: /DataSources/<tenant>/<datasource>.yaml or /DataSources/<datasource>.yaml.

Tenant-specific data sources — file under /DataSources/<tenant>/. Available only to that tenant.
Global data sources — file directly under /DataSources/ (tenant *). These data sources are copied to every tenant. Parameters like {{tenant}} are substituted per tenant.

Only declarations with a discover section are loaded for Chart (Discover/Explore UI).

Elasticsearch¶

/DataSources/events.yaml

define:
  type: datasource/elasticsearch

discover:
  name: Events
  order: 100
  resources:
    - "lmio:dataset:events"

specification:
  index: "lmio-{{tenant}}-events-*"
  schema: /Schemas/ECS.yaml  # optional; defaults to tenant schema
  restrict:                    # optional static filter on every query
    event.dataset: linux

/DataSources/<tenant>/fortinet-fortigate-1.yaml

define:
  type: datasource/elasticsearch

discover:
  name: Fortinet Fortigate 1
  order: 1100

specification:
  index: "lmio-{{tenant}}-fortinet-fortigate-1-*"
  schema: /Schemas/ECS.yaml

`define`¶

type — must be datasource/elasticsearch

`discover`¶

name — display label (required)
order — sort order (default 1000)
resources — list of RBAC resources; user must hold at least one. Users with lmio:dataset:readall see all sources.

Custom RBAC resource

Create a custom resource in SeaCat Auth (for example lmio:dataset:fortinet-fortigate-1) and assign it to roles to control access to a specific data source. List it in discover.resources on the data source declaration.

`specification`¶

index — Elasticsearch index pattern; supports {{tenant}}
schema — Library schema path; if omitted, tenant default from ZooKeeper is used; fallback /Schemas/ECS.yaml.
restrict — static field filters applied to every query. Each filter is a key-value pair.
asset_tag — optional; see Events for assets by tag
asset_kinds — optional list used with asset_tag (host, user, service)

Others data source

Data source Others uses a dedicated schema located in the Library at /Schemas/Others.yaml.

Remote cluster index patterns may include the cluster alias, e.g. {{tenant}}:lmio-{{tenant}}-events*. See Elasticsearch remote cluster.

Events for assets by tag¶

Use this when Discover or Chart should show only events whose principals belong to assets that carry a given tag (for example all hosts with label=prod). Elasticsearch cannot join the asset tag sidecar in MongoDB, so Chart asks LogMan.io Assets for a ready Discover filter and applies it on every query.

How it works

You set specification.asset_tag on an Elasticsearch data source (or pass prompt.asset_tag on a single query).
Chart calls Assets GET /{tenant}/asset/events-filter with that tag (and optional kinds).
Assets resolves matching assets and returns filter language such as host.id:(web01, web02).
Chart AND combines that string into prompt.filters before the Elasticsearch search.

This is the same pattern as resolving an alert ticket into event ids, but for tags instead of tickets.

Permissions

The Chart service must reach lmio-assets over Discovery. The caller needs lmio:asset:access so the events filter endpoint can run.

Library example (hosts with label=prod)

Common Library ships a ready declaration you can copy or adapt:

/DataSources/events_assets_tag_label_prod.yaml

define:
  type: datasource/elasticsearch

discover:
  name: Events (hosts tag=prod)
  order: 150

specification:
  index: lmio-{{tenant}}-events-*
  # UI chip under key "tags". For API key/value tags use "label:prod".
  asset_tag: "prod"
  asset_kinds:
    - host

asset_tag forms

Bare UI chip name (most common): prod → field tags, value prod. The Assets web UI stores chips as a string list under key tags.
Key/value form field:value, for example label:prod → key label = prod (API or activity aux tags). If that key is missing, Assets also matches a UI chip whose text is exactly label:prod.
Mapping form: asset_tag: { field: label, value: prod }.

asset_kinds

Optional list of host, user, and/or service. Each kind maps to one ECS field:

Kind	Event field in the filter
`host`	`host.id`
`user`	`user.id`
`service`	`service.id`

Omit asset_kinds to include all three. Keep only host when you want logs for tagged hosts alone.

Ad hoc query

Without a dedicated data source, set prompt.asset_tag (same field:value string) on a Chart or Discover query against es.events. Data source asset_kinds still apply when present; otherwise all kinds are used.

Empty result

If no asset matches the tag, Chart injects a filter that matches no documents so you do not accidentally see the full event stream.

Related

Analyst UI: Asset management: Tags and events in Discover
Assets service overview: LogMan.io Assets

InfluxDB¶

/DataSources/metrics/cpu.yaml

define:
  type: datasource/influxdb

discover:
  name: CPU
  order: 200

specification:
  measurement: cpu
  restrict:
    host: "{{tenant}}"

`define`¶

type — must be datasource/influxdb

`discover`¶

Same as Elasticsearch (name, order, resources).

`specification`¶

measurement — InfluxDB measurement name
restrict — static tag/field filters; supports {{tenant}} substitution

Schema (tags and fields) is built automatically from SHOW TAG KEYS / SHOW FIELD KEYS on the measurement.

DuckDB¶

Built-in per tenant; not declared in the Library.

DuckDB sources expose derived operational metrics computed in memory from InfluxDB commlink data. Use them for platform monitoring charts (ingest throughput and EPS statistics) without storing extra data in Elasticsearch.

ID	Label	Description
`db.ingest`	Ingest	Ingestion rates (`bytes.in`, `event.in`) per stream and partition
`db.eps`	EPS	Events-per-second aggregates (min, max, mean, median, 95th percentile)

Template variables¶

{{tenant}} — replaced with the current tenant identifier in specification.index, specification.restrict, and InfluxDB restrict values.

Schema resolution¶

For Elasticsearch sources, schema is resolved in order:

specification.schema from the data source declaration
Tenant default schema from ZooKeeper (/lmio/tenants/{tenant}.yaml)
Fallback schema is /Schemas/ECS.yaml

Authorization¶

Data source list and queries are filtered by RBAC. Configure discover.resources on Library declarations, or grant lmio:dataset:readall for unrestricted access.

Data sources¶

Discovery¶

API properties¶

Library layout¶

Elasticsearch¶

define¶

discover¶

specification¶

Events for assets by tag¶

InfluxDB¶

define¶

discover¶

specification¶

DuckDB¶

Template variables¶

Schema resolution¶

Authorization¶

`define`¶

`discover`¶

`specification`¶

`define`¶

`discover`¶

`specification`¶