Skip to content

LMIO Watcher (HTTP REST API)

This document describes the REST API exposed by lmio-watcher for managing lookup definitions and lookup items (content stored in Elasticsearch, index pattern {lookup_id}.lkp).

All paths are prefixed with a tenant segment: /{tenant}/.... Example base URL: http://localhost:8080 (see your deployment / Docker port mapping).

Authentication and authorization

Endpoints use ASAB web auth. Typical resource scopes:

Scope Typical use
lmio:lookup:access Read lookups and items (GET).
lmio:lookup:edit Create/update/delete items, bulk CSV upload, lookup lifecycle edits that change data.
lmio:lookup:global:edit Create or modify global lookups (not tenant-suffixed).

Common conventions

  • Success for simple operations: {"result": "OK"}.
  • Failure (application-level): {"result": "FAIL", "error": "<message>"}.
  • Tenant access: {"result": "TENANT-NO-ACCESS"} with HTTP 401 if the client is not allowed to use the given lookup_id for that tenant.
  • Pagination (where supported): p = page number (1-based), i = page size (default often 10000).
  • Free-text filter (where supported): query parameter f — case-insensitive substring match (see per-endpoint details).

Lookup ID

The lookup_id string identifies a lookup everywhere in URLs, Kafka messages, and Elasticsearch (index {lookup_id}.lkp).

When a lookup is created via POST /{tenant}/lookup-control, the effective id is derived from the JSON field name and whether the lookup is global:

Lookup kind How lookup_id is formed
Tenant-specific (global is false or omitted) {name}.{tenant} — the lookup name (as provided in the request body), a dot, and the tenant taken from the URL path /{tenant}/.... Example: name: "countries" and path .../standard/... → lookup_id = countries.standard.
Global (global: true) {name} only — the value of name is used as the full lookup_id (no .tenant suffix). Creating or editing global lookups requires the lmio:lookup:global:edit scope.

For all HTTP routes, you pass this full lookup_id in the path (e.g. GET /{tenant}/lookup/countries.standard/...), not the bare name alone, when addressing a tenant lookup.

Lookup control (metadata)

Base path: /{tenant}/lookup-control

These endpoints manage lookup declarations (schema, labels, groups, etc.), not the bulk row data (use lookup content below for items).

Method Path Description
GET /{tenant}/lookup-control List lookup definitions available to the tenant. Query: p, i, f (filter matches id, label, name, group, description_title, description_content, default_expiration). Response: {"data": [...], "count": N}.
GET /{tenant}/lookup-control/{lookup_id} Get one lookup definition and schema.
POST /{tenant}/lookup-control Create a lookup. JSON body (validated against lookup create schema): name, label, group, type, global, keys, fields, descriptionTitle, descriptionContent, default_expiration, feedPath, etc. Tenant lookups use id {name}.{tenant} unless global is true.
PUT /{tenant}/lookup-control/{lookup_id} Update lookup declaration.
DELETE /{tenant}/lookup-control/{lookup_id} Delete the lookup.
DELETE /{tenant}/lookup-control/{lookup_id}/items Clear all items in the lookup index (empty content, declaration remains).

Scopes: GET requires lmio:lookup:access; mutating operations require lmio:lookup:edit (and lmio:lookup:global:edit for global lookups where applicable).

Lookup content (items)

Base path: /{tenant}/lookup/{lookup_id}

These endpoints operate on items (documents) inside the Elasticsearch lookup index.

Method Path Description
GET /{tenant}/lookup/{lookup_id} List items. Query: p, i, f (substring search across keys, item id, and declared text-like fields), s<field>=a asc / other values desc for sorting (_id maps to _item_id internally). Response: {"data": [...], "count": N}.
GET /{tenant}/lookup/{lookup_id}/by List items where field = value (exact term). Required query: field, value. Same pagination/sort/filter p, i, s*, f as list.
GET /{tenant}/lookup/{lookup_id}/{_id} Get one item by document id (URL-encoded; / in ids uses %2F).
POST or PUT /{tenant}/lookup/{lookup_id} Create one item. JSON: keys / fields arrays (UI shape), optional _exp.
PUT /{tenant}/lookup/{lookup_id}/{_id} Update one item. Same optional keys as create when changing key values (may re-index under a new id).
DELETE /{tenant}/lookup/{lookup_id}/{_id} Delete one item.

Scopes: read endpoints need lmio:lookup:access; write/delete need lmio:lookup:edit (and global edit for global lookups).

Bulk CSV upload (highlight)

The primary way to load or refresh many rows at once is CSV multipart upload. Internally, rows are applied via bulk update (update_by_bulk) against Elasticsearch.

Endpoints (equivalent)

Method Path
POST /{tenant}/lookup/{lookup_id}/upload
POST /{tenant}/lookup/{lookup_id}/import

Both invoke the same handler. Pick whichever fits your tooling; /upload is the conventional name for UI “bulk import”.

Authorization

Requires lmio:lookup:edit (and global rules apply for global lookups).

Request

  • Content-Type: multipart/form-data
  • Body: one file part (first file in the multipart stream is used). The file is read from disk under /tmp/<original_filename> (ensure filename is unique under concurrency in production).
  • Query parameters:
  • delimiter — CSV field delimiter. Default is ; (semicolon). Use delimiter=, for comma-separated files.

CSV shape

  • First row must be a header (csv.DictReader).
  • Columns:
  • _id — optional; document id if present.
  • _keys — optional; for compound keys, comma-separated parts in one cell (split on commas after strip).
  • _exp — optional expiration (string cell, stripped).
  • Declared field names — any column whose name matches the lookup declaration fields map is imported; others are ignored.

Only fields present in the declaration are copied; geopoint columns accept either DMS “lat, lon” pairs or a numeric SPL geopoint value (same rules as single-item create).

Response

{
  "result": "OK",
  "count": 1234
}

count is the number of documents reported by the bulk update operation.

Errors

Condition Typical response
No file in multipart {"result": "NO-FILE-UPLOADED"} with HTTP 400
No access to lookup {"result": "TENANT-NO-ACCESS"} with HTTP 401

Example (curl)

curl -X POST \
  -H "Content-Type: multipart/form-data" \
  -F "file=@countries.csv" \
  "https://<host>:<port>/<tenant>/lookup/my_lookup.<tenant>/upload"

With comma-separated CSV:

curl -X POST \
  -F "file=@data.csv" \
  "https://<host>:<port>/<tenant>/lookup/my_lookup.<tenant>/upload?delimiter=,"

CSV export and sample

Method Path Description
GET /{tenant}/lookup/{lookup_id}/export Download all items as CSV (Content-Type: text/csv, attachment). Header includes _id, optionally _keys for compound-key lookups, then declared fields. Semicolon-separated; geopoint exported as DMS when applicable.
GET /{tenant}/lookup/{lookup_id}/export_sample JSON response: {"result": "OK", "data": "<csv string>"} — first page of rows (up to 5 items) in the same CSV format for preview.

Scope: lmio:lookup:access.

Notes for API clients

  1. Lookup id often ends with .{tenant} for tenant-specific lookups; global lookups use ids without that suffix.
  2. Item _id in URLs must be percent-encoded when it contains special characters (e.g. / → %2F).
  3. List filter f on items searches normalized text across keys, _item_id, and declared string-like / MAC-related fields; numeric-only fields may not match substring search.
  4. Bulk upload does not replace the whole index by itself—it upserts each parsed row through the bulk path; use clear items (DELETE .../lookup-control/{lookup_id}/items) first if you need an empty lookup before a full reload.