Lookup types and declarations¶

This page describes how lookups are classified in LogMan.io: declaration shape, built-in lookup kinds (define.type), simple vs compound (multi-part) keys, key and field types, and how lookup items are stored when you edit them in the product.

`lookup/compoundkey`, `lookup/string`, and generic `lookup`¶

When you create or edit a lookup in LogMan.io, the declaration stores a type that controls how many keys the schema has:

Declaration `type`	Meaning
`lookup/string`	Default for a new lookup: one key, usually text (`str`). Behaviour matches a generic lookup with a single string key. In the Library, the same idea is often written as `define.type: lookup` with one `str` key.
`lookup/compoundkey`	Several key parts (you can add more keys in the schema). Behaviour matches a generic `lookup` with compound keys: same storage rules, composite keys are built in canonical form and hashed (see below). `compoundkey` is a type name for “generic lookup with more than one key part”.
`lookup/number`	One key of type `ui64` (unsigned integer). Same storage pattern as other generic single-key lookups—typically a `.bin` data file.

lookup/compoundkey is a normal declaration value. You can also define multi-key lookups in the Library as define.type: lookup with multiple keys:; the behaviour is the same family as lookup/compoundkey.

Lookup kinds (`define.type`)¶

The implementation is determined by define.type in the declaration—whether the file lives under /Lookups/ in the Library or was created when managing lookups in LogMan.io.

`define.type` (prefix match)	Role	How it is stored / resolved
`lookup` (generic)	Key → record (`.bin` / `.ip` / `.mac`, or loaded from Elasticsearch / other backing store as configured).	Fast local files and/or remote fill of the cache, depending on setup.
`lookup/string`	Single text (`str`) key.	Resolves like generic with string key → `{lookup_id}.bin`
`lookup/compoundkey`	Multiple keys (compound).	Same as generic multi-key → `{lookup_id}.bin` + canonical composite key
`lookup/number`	Single `ui64` key.	Same as generic → `{lookup_id}.bin`
`lookup/ipaddress`	Single exact IP key → record (one `ip` key).	`{lookup_id}.ip`
`lookup/macaddress`	Single exact MAC key → record (one `mac` key).	`{lookup_id}.mac`
`lookup/ipaddressrange`	Private IP range → record.	Dedicated range format, file `*.ipr`
`lookup/ipaddressrangegeo`	Public / Geo IP range → record.	Dedicated geo-range format, file `*.iprgeo`
`lookup/macaddressrange`	MAC range → record.	Dedicated range format, file `*.macr`
`lookup/maxmind`	MaxMind database lookup.	External `.mmdb` file

Range and MaxMind lookups use special file formats and are usually not maintained as simple row-by-row lists in the same way as tenant lookups whose items live in *.lkp indices. Their declarations (metadata, labels) can still be registered for a tenant.

For examples of generic, IP, and MAC declarations, see Lookups index, Generic lookups, IP address lookups, and MAC address lookups.

Generic lookup: data file suffix¶

For the generic lookup family—including lookup/string, lookup/compoundkey, lookup/number, and lookup/ipaddress / lookup/macaddress—the suffix of the on-disk file depends on the effective key type after the keys are known:

Effective key type	File suffix
`ip`	`{lookup_id}.ip`
`mac`	`{lookup_id}.mac`
Anything else (e.g. `str`, compound tuple)	`{lookup_id}.bin`

Keys: simple vs compound¶

Simple key¶

keys: has one entry, for example:

keys:
  - name: user.id
    type: str

The system treats the lookup as having that single key type: str, ip, mac, ui64, and so on.

Compound key (“multi-part” lookup)¶

When you choose compound key as the lookup type in LogMan.io, or define lookup with several keys: in the Library, you get a multi-part key—for example IP + username:

keys:
  - name: source.ip
    type: ip
  - name: user.name
    type: str

The key type is described as a combination such as (ip,str,).

At runtime, values for all key parts are turned into a canonical form, then a 64-bit xxHash of compact JSON (separators=(',', ':')) is computed and stored as a hex string. That string is what the lookup uses internally (and it must stay consistent when data is built from Elasticsearch exports or other dumps). The binary store treats it as a string key.

IP parts in compound keys use the JSON IPv6 encoding {"h":...,"l":...} internally (not only the dotted form) so encoding stays stable.

Using a compound key in rules (`!TUPLE`)¶

In SP-Lang—correlator predicates, membership checks against a lookup, lookup actions on triggers (including delete)—you must pass the composite key as a !TUPLE. Elements must appear in the same order as in the lookup declaration’s keys: list, with matching types (ip, str, and so on).

If an event field can contain several values (for example host.ip as a list), take one element with !GET (for example index 0) so that key part is a single value.

Example: lookup servicedown-ttl15m with two key parts (here host.ip then service.name—align this with your declaration):

predicate:
  !AND
  - !IN
    what: !ITEM EVENT event.code
    where: ["7036", "7040"]
  - !IN
    what: "started"
    where: !ITEM EVENT message
  - !IN
    what: service.name
    where: !EVENT
  - !IN
    what:
      !TUPLE
      - !GET
        what: 0
        from: !ITEM EVENT host.ip
      - !ITEM EVENT service.name
    where:
      !LOOKUP
      what: servicedown-ttl15m

evaluate:
  dimension: [host.ip, service.name]
  by: "@timestamp"
  resolution: 60
  saturation: 1

analyze:
  window: hopping
  aggregate: sum
  span: 2
  test:
    !GE
    - !ARG
    - 1

trigger:
  - lookup: servicedown-ttl15m
    delete:
      !TUPLE
      - !GET
        what: 0
        from: !ITEM EVENT host.ip
      - !ITEM EVENT service.name

The !IN sub-expression tests that the tuple key exists in the lookup; delete on the lookup trigger removes the row for that same tuple.

Record `fields`: supported types (conceptual)¶

Values in each lookup item are stored in a compact binary layout. Typical field types include:

Type	Meaning (high level)
`str`	UTF-8 text
`ip`	IPv4/IPv6 in a fixed binary layout
`mac`	48-bit MAC
`ui16`, `ui64`, `si32`, `si64`, …	Fixed-width integers
`fp64`	`double`
`bool`	0/1
`geopoint`	Geopoint as `uint64` (lat/lon fixed-point)
`datetime`	Datetime with conversions suitable for Elasticsearch
`[str]`, `[ip]`, `[ui64]`, `[ui16]`, `[mac]`, `[si64]`, …	Lists with the corresponding element layout

Field names should be flat keys in the declaration—not nested ECS-style paths. The same flat names appear in APIs and in stored documents.

Storing editable lookup items (Elasticsearch)¶

For lookups whose items you edit in LogMan.io (lists of rows):

Data lives in an Elasticsearch index named {lookup_id}.lkp (lowercased).
Each document has _id, _keys, optional _item_id (for sorting), versioning / timestamps (_c, _m, _lv, _exp, …), and attributes that match the declaration fields.

When items are saved, values are normalised to Elasticsearch field types (IPs, MACs, geopoints, integers, floats, etc.) so _id and _keys stay consistent with how lookups are resolved during processing.

List / search filter (`f`)¶

When listing or searching items, a free-text parameter f may be available:

Search is usually a case-insensitive substring across _keys, _item_id, and fields whose types support text wildcard search (for example str, [str], mac, macrange, and similar list forms).
Fields stored as plain numbers, IP, geo, or other non-keyword types are not included in that wildcard search (Elasticsearch cannot run substring wildcard on arbitrary numeric field types).
For MAC-style keys or fields, hyphen - in f is often normalised to : so common MAC input matches stored values.

Fast local file vs on-demand load¶

Mode	Data source	Typical use
Local mmap file	Pre-built binary file on disk	Very fast reads during processing
Backed by Elasticsearch (and optionally other stores)	Rows fetched when missing from cache	Fills cache on demand; used when the file is absent or in hybrid setups

If the local file is missing or unusable, behaviour may fall back to loading from the remote store, depending on how LogMan.io is deployed.

Lookup kinds (short summary)¶

Generic — lookup, lookup/string, lookup/compoundkey, lookup/number: string / number / compound-hash keys; .bin (or .ip/.mac for single IP/MAC key types where applicable).
Exact IP / MAC — lookup/ipaddress, lookup/macaddress: one exact key; file suffix by key type.
IP range — lookup/ipaddressrange.
IP range + geo — lookup/ipaddressrangegeo.
MAC range — lookup/macaddressrange.
MaxMind — lookup/maxmind; .mmdb database file.

Type string in rules¶

In advanced usage, a lookup may be referred to by a type string such as:

lookup<(ip,str):(username:str,score:ui64)>

Compound key types appear in the first angle-bracket group as (type1,type2,...).