Mapping declarations

After all declared fields are obtained from parsers, the fields have to be renamed according to some schema (ECS, CEF, ...) in a process called mapping. Mapping ensures that logs from various sources have unified, consistent field names and types.

The mapping process:

1. renames the fields of the parsed logs according to some schema
2. eventually converts field types (e.g. from string to integer, IP, MAC etc.)
3. filters out all fields that are not listed in mapping

Declaration

define:
    type: parsec/mapping
    schema: /Schemas/ECS.yaml

mapping:
    <original_key>: <new_key>
    <original_key>: <new_key>
    ...

Specify parsec/mapping as the type in the define section. In the schema field, specify the path to the schema you're using.

Example

For the purpose of the example, let's say that we want to parse a simple event:

User harry_potter login from 178.2.1.20

and we would like the final output look like this:

{
    "user.name": "harry_potter",
    "event.action": "login",
    "source.ip": "178.2.1.20"
}

Notice that the key names in the original event differ from the key names in the desired output.

For the initial parser declaration in this case, we can use a simple JSON parser:

10_parser.yaml

define:
    type: parsec/parser

parse:
    !PARSE.KVLIST
    - "User "
    - USER: !PARSE.UNTIL " "
    - ACTION: !PARSE.UNTIL " "
    - !PARSE.UNTIL " "
    - IP: !PARSE.IP

This parser will create a list of key-value pairs:

USER        harry_potter
ACTION      login
IP          178.2.1.20

To change the names of individual fields, we create mapping declaration file, 20_mapping_ECS.yaml, in which we describe what fields to map and how:

20_mapping_ECS.yaml

---
define:
    type: parsec/mapping  # determine the type of declaration
    schema: /Schemas/ECS.yaml  # which schema is applied

mapping:
    USER: user.name
    ACTION: event.action
    IP: source.ip

This declaration will produce the desired output.