Skip to content

Parsing

Parsing is the process of analyzing the original log (which is typically in single/multiple-line string, JSON, or XML format) and transforming it into a list of key-value pairs that describe the log data (such as when the original event happened, the priority and severity of the log, information about the process that created the log, etc).

Every log that enters your TeskaLabs LogMan.io system needs to be parsed. The LogMan.io Parsec microservice is responsible for parsing logs. The Parsec needs parsers, which are sets of declarations (YAML files) to know how to parse each type of log. LogMan.io comes with the LogMan.io Common Library, which has many parsers already created for many common log types. However, if you need to create your own parsers, understanding parsing key terms, learning about declarations, and using the parsing tutorial can help.

Basic parsing example

Parsing takes a raw log, such as this: 

<30>2023:12:04-15:33:59 hostname3 ulogd[1620]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60002" initf="eth2.3009" outitf="eth6" srcmac="e0:63:da:73:bb:3e" dstmac="7c:5a:1c:4c:da:0a" srcip="172.60.91.60" dstip="192.168.99.121" proto="17" length="168" tos="0x00" prec="0x00" ttl="63" srcport="47100" dstport="12017"
And sorts and separates it into fields that are easier to read, understand, and filter:

# Timestamps
'@timestamp': 2023-12-04 15:33:59.033
event.created: 2023-12-04 15:33:59.033
event.ingested: 2023-12-04 15:39:10.039

# Syslog header
log.syslog.priority: 30
log.syslog.facility.code: 3
log.syslog.facility.name: daemon
log.syslog.severity.code: 6
log.syslog.severity.name: information

host.hostname: hostname3
process.name: ulogd
process.pid: 1620

# Event fields
event.dataset: sophos
event.action: Packet dropped
event.id: 2001

# Source fields
source.bytes: 168
source.ip: 172.60.91.60
source.mac: e0:63:da:73:bb:3e
source.port: 47100

# Destination fields
destination.ip: 192.168.99.121
destination.mac: 7c:5a:1c:4c:da:0a
destination.port: 12017

# Other relevant information
observer.egress.interface.name: eth6
observer.ingress.interface.name: eth2.3009
sophos.action: drop
sophos.fw.rule.id: 60002
sophos.prec: 0x00
sophos.protocol: 17
sophos.sub: packetfilter
sophos.sys: SecureNet
sophos.tos: 0x00
device.model.identifier: SG230
dns.answers.ttl: 63