Skip to content

Library in TeskaLabs LogMan.io

Library contains declarations for parsers, enrichers, correlators, templates for emails and instant messages, dashboards, reports, and other content of TeskaLabs LogMan.io.

Library organization

Library is organized into folders with items, similar to the files on the file system.

The following example illustrates Library organization with three base folders /Baselines/, /Dashboards and /Parsers/:

/Baselines/
    Dataset.yaml
    Host.yaml
    User.yaml

/Dashboards/
    Linux/
        Common/
            Overview.json

/Parsers/
    Linux/
        Auditd/
            10_parser.yaml
            20_enricher.yaml
            30_mapping_ECS.yaml
        Common/
            10_parser.yaml
            20_parser_process.yaml
            30_parser_message.yaml

Library path rules

Internally (in configuration of micro-services etc.), Library paths must satisfy these rules:

  • Every path MUST start with "/", including the root path. Only absolute paths are used, e.g. /Parsers/Microsoft/Exchange/.
  • A folder path MUST end with "/", e.g. /Parsers/Microsoft/Exchange/.
  • An item path MUST end with an extension (e.g. ".txt", ".json", ...), e.g. /Parsers/Microsoft/Exchange/10_parser.yaml
  • A folder name MUST NOT contain ".". An item name MUST NOT start with ".".

Note

A user cannot create new folders at the topmost / folder. Some folders do allow a user to add/edit/delete files and folders inside them, while some do not. Each Library folder has its own rules for that.

Some Library folders allow only specific file extensions.

For example: The folder /Parsers/ allows creating new folders and parsing rules ending with .yaml inside.

The folder /Homepage/ does not allow to add more items (as there is only one homepage).

The folder /Alerts/Workflow/ does not allow creating new folders inside while allowing adding or deleting the existing alert workflows.

Library layers

Library is organized into layers. Each Library layer refers to a single source of Library files and particular storage technology. Library layers are "stacked" into one view (overlayed), merging the content of each layer into one united space. This layering allows to combine a content from different sources (aka providers) into one Library. Library layers are initially configured in the product during the installation. There is no limit to the number of stacked library layers.

Library layers

Schema: An example of library layers setup, the User will see items in a green box.

The example of configuration implementing the above schema.

The library is configured either centrally using ASAB Maestro or in the each microservice configuration file.

[library]
providers:
    zk://
    libsreg+https://libsreg1.example.com,libsreg2.example.com/my-library#v24.11
    git+https://github.com/john/awesome_project.git#deployment
    libsreg+https://vendor.example.com/common-library
  • Layer 0 is in Apache ZooKeeper, reusing the ZooKeeper configuration from [zookeeper] section.
  • Layer 1 is a "my-library" distributed using libraries registry.
  • Layer 2 is from a Git (or GitHub more specifically), tracking "deployment" branch
  • Layer 3 is a Vendor "common library".

Writable layer

The layer 0 is the only writable layer. It is intended to create and edit custom content. A user can edit the content of this layer using the "Library" editor in the User Interface.

Higher layers are read-only, meant for the one-directional distribution of the Library content to deployments.

If a user edits an item that is present in the read-only layers, it is stored in the layer 0, overriding the item of the same path on higher layers. This is the mechanism for how users can modify the out-of-the-box content provided by Common libraries and so on.

The layer 0 is further divided into targets, this could be a global target for content that is available for the whole deployment or a tenant target which stores the content for specific tenants. It means the tenant-specific content such as Parsers or Dashboards is stored in Layer 0, in the target "tenant".

Library layer types

Each library layer is delivered through provider.

Apache ZooKeeper

Layer stored locally in the Apache ZooKeeper technology. It is a distributed and redundant layer through a consensus mechanism in ZooKeeper. This layer is a typical choice for writable Layer 0.

The configuration prefix: zk:/.

Note

Administrators can also use the ZooNavigator tool to explore and modify the content of a ZooKeeper layer.

Libraries repository

Content in this layer is provided through a distribution point specified by a URI. A distribution point is a server or a public cloud storage accessible over HTTPS. It is a read-only layer. The distribution over the library's repository is the preferred way of content distribution. The content of the layer is automatically refreshed, so that if the updated content is available at the libraries repository servers, it will be distributed into libraries in deployments.

The configuration prefix: libsreg:/.

Example: libsreg+https://libsreg1.example.com/my-library.

Versioning

This provider supports versioning. The version is specified in the fragment part of the URL (after # symbol):

Example: libsreg+https://libsreg1.example.com/my-library#v24.11

Versions can be static (i.e. v24.11, not changing after its release) or rolling (i.e. production or main, changes are continuously propagated into a distribution point and hence to deployments).

Tip

The versioning is designed to work with CI/CD process on the library content publisher. Typically, the "main" or golden copy of the library is stored in Git on the CI/CD platform and it is deployed to the distribution point by CI/CD automation.

Resiliency

This provider supports resilient content delivery. You can specify more than one distribution server in the configuration. The TeskaLabs LogMan.io will iterate to other specified servers if the request fails.

Example: libsreg+https://libsreg1.example.com,libsreg2.example.com/my-library

Git

The layer that provides a (read-only) content from a Git repository. It is meant for a continuous delivery of the content from the Git server such as GitHub or GitLab.

The configuration prefixes: git+https:/, git+http:/ or git:/.

Versioning in Git

This provider supports versioning. The version is specified in the fragment part of the URL (after # symbol):

Example: git+https://github.com/john/awesome_project.git#deployment

The default version is set in the particular Git repository. The typical values are master or main. It is used when no fragment is provided.

Versions can be static (i.e. v24.11, not changing after its release) or rolling (i.e. production or main, changes are continuously propagated into a distribution point and hence to deployments).

Filesystem

Layer stored at the filesystem.

Warning

Since the filesystem is local to the node, this layer type is not suitable for use in clusters.

The configuration prefixes: file:/ or /.. (as in absolute filesystem paths).

Microsoft Azure Storage

Layer that provides a (read-only) content from a container located at Microsoft Azure Storage.

The configuration prefix: azure+https:/

Note

If the Container Public Access Level is not set to "Public access", then "Access Policy" must be created with "Read" and "List" permissions and "Shared Access Signature" (SAS) query string must be added to a URL in a configuration:

[library]
providers: azure+https://ACCOUNT-NAME.blob.core.windows.net/BLOB-CONTAINER?sv=2020-10-02&si=XXXX&sr=c&sig=XXXXXXXXXXXXXX

Enabling & disabling the content

Any item of the Library content can be disabled globally or specifically for a tenant. By disabling files, administrators can modify content accessible by users of particular tenant.

The content can be enabled/disabled from the "Library" screen or using /.disabled.yaml file located at the layer 0.

LogMan.io Common Library

LogMan.io Common Library is a distributed content that is located at the highest layers of Library. It represents a default content, provided by the TeskaLabs or partners.

Full-text indexing

The content of the library (all layers) is automatically indexed so that users can quickly search for specific content.

More info

The technical details can be also found here https://docs.teskalabs.com/asab.