LogMan.io Tenant Rex

LogMan.io Tenant Rex (lmio-trex) is the microservice that owns business tenants and automatically deploys per tenant baseliner and correlator Remote Control models.

This page is the single place for operators and admins who need to understand how T Rex places correlators, recovers them, and which files it cleans up.

Analysts who only enable rules in the UI should read Detection rule deployment.

What T Rex does

1. Creates, updates, and removes business tenant objects (MongoDB), including attributes such as schema and timezone.
2. Writes Library models at /Site/model-trex-{tenant}.yaml for lmio-baseliner and lmio-correlator.
3. Asks ASAB Remote Control to run compose up after model changes (procedures node_upgrade or cluster_sequential_up).
4. Optionally talks to Kafka Admin APIs for consumer lag, membership (HA), and deletion of unused correlator consumer groups.

T Rex does not evaluate correlation rules. Rule evaluation stays inside lmio-correlator.

Leader election

With ZooKeeper leader election ([trex] leader_name, default lmio-trex):

1. Only the leader rebuilds models, runs correlator lifecycle (lag / HA / Kafka housekeeping), and dumps tenants to ZooKeeper.
2. Followers keep the HTTP API.

Run one or more T Rex instances per cluster. They must share MongoDB and ZooKeeper.

Correlator placement

T Rex builds correlator instances from the tenant Detections selection in MongoDB (Library folders; a selected file maps to its parent folder).

Excluded folders

By default these Library prefixes never become correlator groups ([correlator] exclude_folders):

1. /Detections/Parsec/
2. /Detections/Filters/

They are dropped from deploy selections and from model builds.

Folder depth (folder_max_depth)

By default T Rex places a correlator on the nearest / deepest selected folder.

You can cap depth under selected roots with [correlator] folder_max_depth (multiline path=N). Depth is the number of path segments below that root.

Default:

[correlator]
folder_max_depth=
    /Detections/Sigma/=3

Example with /Detections/Sigma/=3:

1. Rule path: /Detections/Sigma/windows/builtin/security/account_management/rule.yaml
2. Correlator groups: /Detections/Sigma/windows/builtin/security/
3. Deeper subfolders share that one instance (the correlator still loads rules recursively).

Paths that are not listed keep nearest / deepest folder behaviour (typical for /Correlations/…).

Setting folder_max_depth in config replaces the whole default list. Include /Detections/Sigma/=3 again if you add other roots and still want the Sigma cap.

Empty or missing folders

If a selected folder has no enabled rule files in the Library (empty, deleted, or all disabled), T Rex omits that correlator from the model. The next rebuild undeploys it via Remote Control up.

Kafka consumer group_id

Every correlator instance gets an explicit Kafka consumer group under pipeline:CorrelatorsPipeline:KafkaSource:

lmio_correlator_<instance_id>

Dashes in the instance id become underscores.

When an instance leaves the model, T Rex deletes that consumer group (if [kafka] bootstrap_servers is set). Periodic housekeeping also removes orphan groups with prefixes lmio_correlator* and legacy lmio-correlator*.

Disk data directories are not deleted. After an instance is removed, a host folder such as /data/ssd/correlators/lmio-correlator-… may remain on the node. Remote Control up does not wipe unused volume sources. Clean those up on the node if needed.

Lag based rule splitting

When [correlator:lag] enabled=yes and Kafka is reachable:

1. T Rex samples consumer lag for known groups.
2. Chronically rising lag lowers the effective rules per instance limit for that folder.
3. The next model rebuild creates more instances with the same groups folder but different read_from / read_declarations slices.
4. Each slice has its own group_id and still reads the full event stream; it only evaluates its rule subset.

This adds CPU capacity for rules. It is not Kafka partition scaling.

High availability (HA)

When [correlator:ha] enabled=yes and Kafka is reachable:

1. T Rex watches Kafka member count for each correlator group.
2. If members stay at zero longer than unhealthy_after_sec (default 300 s), T Rex requests Remote Control up on the current node (local restart; placement unchanged).
3. If members are still zero for another unhealthy_after_sec after that local up, T Rex pins the instance to a different eligible node (ZooKeeper nodes excluding arbiter and no-trex) and uploads the model again.

failover_cooldown_sec (default 600 s) limits how often a move can repeat.

When failover is skipped

Node failover cannot fix a broken correlation rule. T Rex therefore does not move the instance when ASAB discovery for that correlator still reports:

1. correlation_rules_with_errors (from correlator AttentionService)
2. or attention_required on the microservice advertisement

In that case you get at most one local up, then a warning to fix rules / library problems. Check correlator logs and Library attentions.

Circuit breaker

[correlator:ha] max_failovers (default 2) caps how many node moves happen in one unhealthy episode. After that, T Rex stops failing over (useful for crash loops where discovery never advertises). Set 0 for unlimited moves (not recommended).

Active active placement

When [correlator:active_active] enabled=yes (default off):

1. Each logical correlator slice is deployed on every eligible node.
2. Replicas share the same config and the same Kafka group_id.
3. Node move failover is disabled; local up can still run if the shared group has zero members.

Baseliner placement

By default ([baseliner:ha] enabled=yes) T Rex places one baseliner instance on every eligible node. Set enabled=no for a single hash placed instance per tenant.

Configuration reference (lmio-trex)

[kafka]
# Required for lag, HA membership, and consumer group deletion.
bootstrap_servers=kafka-1:9092,kafka-2:9092

[correlator]
exclude_folders=
    /Detections/Parsec/
    /Detections/Filters/
folder_max_depth=
    /Detections/Sigma/=3

[correlator:lag]
enabled=yes
sample_every_ticks=6
housekeep_every_ticks=30

[correlator:ha]
enabled=yes
unhealthy_after_sec=300
failover_cooldown_sec=600
max_failovers=2

[correlator:active_active]
enabled=no

[baseliner:ha]
enabled=yes

[trex]
leader_name=lmio-trex

[asab:remote:control]
# Wait for Remote Control procedure completion (seconds). 0 = fire and forget after Accepted.
procedure_wait_sec=600
procedure_poll_sec=5

HTTP API listen port: 8962 (see Network ports).

Related reading

1. Detection rule deployment (analyst UI)
2. Monitoring detection performance (lag and slow rules)
3. Correlator configuration (manual models and Kafka for lmio-correlator)
4. Correlator metrics