Parsing

Parsing is the process of analyzing the original log (which is typically in single/multiple-line string, JSON, or XML format) and transforming it into a list of key-value pairs that describe the log data (such as when the original event happened, the priority and severity of the log, information about the process that created the log, etc).

Every log that enters your TeskaLabs LogMan.io system needs to be parsed. The LogMan.io Parsec microservice is responsible for parsing logs. The Parsec needs parsers, which are sets of declarations (YAML files) to know how to parse each type of log.

All parsed fields in LogMan.io are mapped to some schema. This ensures that log data from different sources is normalized and structured consistently, making it easier to search, analyze, and correlate events across your environment.

LogMan.io comes with the Common Library, which has many parsers already created for many common log types. However, if you need to create your own parsers, understanding parsing key terms, learning about declarations, and using the parsing tutorial can help.

Basic parsing example

Parsing takes a raw log, such as this (example from Sophos Firewall):

<30>2023-12-04T15:33:59.033+00:00 hostname3 ulogd[1620]: id="2001" severity="info"
sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop"
fwrule="60002" initf="eth2.3009" outitf="eth6" srcmac="e0:63:da:73:bb:3e"
dstmac="7c:5a:1c:4c:da:0a" srcip="172.60.91.60" dstip="192.168.99.121"
proto="17" length="168" tos="0x00" prec="0x00" ttl="63"
srcport="47100" dstport="12017"

And sorts and separates it into fields that are easier to read, understand, and filter:

# Timestamp
"@timestamp": 2023-12-04 15:33:59.033

# Syslog header
log.syslog.priority: 30
log.syslog.facility.code: 3
log.syslog.facility.name: daemon
log.syslog.severity.code: 6
log.syslog.severity.name: information

host.hostname: hostname3
process.name: ulogd
process.pid: 1620

# Event fields
event.dataset: sophos
event.action: Packet dropped
event.id: 2001

# Source fields
source.bytes: 168
source.ip: 172.60.91.60
source.mac: e0:63:da:73:bb:3e
source.port: 47100

# Destination fields
destination.ip: 192.168.99.121
destination.mac: 7c:5a:1c:4c:da:0a
destination.port: 12017

# Other relevant information
observer.egress.interface.name: eth6
observer.ingress.interface.name: eth2.3009
sophos.action: drop
sophos.fw.rule.id: 60002
sophos.prec: 0x00
sophos.protocol: 17
sophos.sub: packetfilter
sophos.sys: SecureNet
sophos.tos: 0x00
device.model.identifier: SG230
dns.answers.ttl: 63