Collecting logs from Syslog¶

To collect logs from various Syslog sources, TeskaLabs LogMan.io Collector provides an option to setup a TCP and/or UDP receival port that conforms to a wide variety of Syslog protocols. You can simply configure a log source (for example Linux server or a network appliance) to send logs to the collector, typically to the port 514 (TCP and/or UDP) and TeskaLabs LogMan.io Collector will ingest these logs.

Tip

Syslog protocol typically uses port 514, TCP or UDP, but TeskaLabs LogMan.io Collector can be configured to use other ports as well.

To utilize this feature, configure your log sources to forward logs to port 514. TeskaLabs LogMan.io Collector will automatically detect the origin of the incoming data and categorize it into streams (event lanes) accordingly.

Smart classification¶

TeskaLabs LogMan.io Collector classifies incoming logs using a smart feature. This enables a convenient ability to configure all log sources to send logs to a TeskaLabs LogMan.io Collector IP address and a specific port, including a very simple network path configuration.

TeskaLabs LogMan.io Collector uses a classification map in its YAML configuration.

Configuration¶

This example shows a configuration of TeskaLabs LogMan.io Collector for using Syslog and a smart classification of the incoming logs on port 514 UDP and TCP.

Stream name is important

A stream name specified in configuration of LogMan.io Collector is important. LogMan.io selects a proper parsing rules, content and other components related to the log source based on the stream name. Read more about stream names here.

Minimal configurationConfiguration with classified streams

Here is an example of a minimal configuration in which all incoming logs will be sent to a stream generic.

classification:
  smart514: &smart514  # YAML anchor referencing to both SmartDatagram and SmartStream inputs

    # All events are sent to stream 'generic'
    generic:
      - ip: "*"

# Listen on the UDP 514
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart514
  output: smart

# Listen on the TCP 514
input:SmartStream:TCP514:
  address: 514
  smart: *smart514
  output: smart

# Connection to LogMan.io
output:CommLink:smart: {}

Here is an example of configuration with four streams linux-syslog-rfc5424-1, fortinet-fortigate-1, fortinet-fortigate-2 and linux-rsyslog-1 with various options of how you can classify the incoming IP addresses, ports and protocols.

classification:

  smart514: &smart514  # YAML anchor referencing to both SmartDatagram and SmartStream inputs

    linux-syslog-rfc5424-1:    # stream name

      - ip: "192.168.0.1"      # Single IPv4 address
        port: 80               # Single port
        protocol: TCP          # TCP protocol

      - ip: "2001:db8::1"      # Single IPv6 address
        port: "1000-2000"      # Port range
        protocol: UDP          # UDP protocol

    fortinet-fortigate-1:

      - ip: "10.0.0.0/8"       # IPv4 range
        port: 14000
        protocol: UDP

      - ip: "fd00::/8"         # IPv6 range
        port: "*"              # Any port
        protocol: UDP

    fortinet-fortigate-2:

      - ip: "*"                # Any IP address
        port: "*"              # Any port
        protocol: UDP

    linux-rsyslog-1:
      - ip: "::1"              # Local IP addresses

# Listen on the UDP 514
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart514
  output: smart

# Listen on the TCP 514
input:SmartStream:TCP514:
  address: 514
  smart: *smart514
  output: smart

# The logs are forwarded to a LogMan.io using a CommLink
output:CommLink:smart: {}

Warning

Smart classification works only with a CommLink output.

SmartDatagram and SmartStream¶

SmartDatagram (for UDP) and SmartStream (for TCP) sources are similar to Datagram/TCP and Stream/UDP sources, with additional option smart, which references the appropriate sub-section in classification.

# Listen on the UDP 514
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart514
  output: smart

# Listen on the TCP 514
input:SmartStream:TCP514:
  address: 514
  smart: *smart514
  output: smart

Warning

Section classification must be specified BEFORE input:... sections, otherwise the reference *smart514 is not recognized.

Classification map¶

Section classification can contain one or more classifiers.

Each classifier specifies a combination of IP address ranges, port ranges and protocols; and it resolves them into a stream. Every log that arrives to the TeskaLabs LogMan.io Collector smart syslog is matched with these classifiers and a resulting stream is used as its destination in LogMan.io. If no match is found, the log goes into generic stream (named generic).

Tip

Streams can be found in the Archive component of TeskaLabs LogMan.io.

linux-syslog-rfc5424-1:
  - ip: "192.168.0.1"
    port: 80
    protocol: TCP

  - ip: "2001:db8::1"
    port: "1000-2000"
    protocol: UDP

linux-syslog-rfc5424-1 is a stream name.

ip¶

A single IPv4/IPv6 address: 92.168.0.1, 2001:db8::1
Range of IPv4/IPv6 addresses: 10.0.0.0/8, fd00::/8
Wildcard * for all IPv4/IPv6 addresses

port¶

A single port: 5400
Port range: 4000-8000
When not specified or *, range 0-65535 is used

protocol¶

TCP / UDP
When not specified, both TCP and UDP are used

* wildcard can go too wild

Be sure to wrap * wildcard into quotation marks in YAML "*". Aterisk without quotation marks would break YAML syntax.

Overlapping IP addresses and ports¶

IP addresses and ports can overlap. In that case, the most specific match is selected. In the example below, 25400 is matched with fortinet-fortigate-3, 25100 with fortinet-fortigate-2 and 24000 with fortinet-fortigate-1:

fortinet-fortigate-1:
  - ip: "192.168.0.1"
    port: 24000-30000

fortinet-fortigate-2:
  - ip: "192.168.0.1"
    port: 25000-26000

fortinet-fortigate-3:
  - ip: "192.168.0.1"
    port: 25400

The same holds for IP addresses.

Generic stream¶

If no target stream is identified during the classification, a log is forwarded to generic stream.

Example of stream classification

Suppose you connect new log sources from IP address range 192.168.0.0/24.

With no classifier, events are collected into generic stream and stored in Archive.

After looking into streams in Archive, you discovered there is a source of type logsox. You classify the stream to separate it from other incoming data:

logsox-1:
  - ip: "192.168.0.0/24"

When creating parsing rules for the stream, you discover that logsox incoming events from IP 192.168.0.68 have a different form than the others. You can isolate that stream and apply different parsing rules for it:

logsox-1:
  - ip: "192.168.0.0/24"

logsox-2:
  - ip: "192.168.0.68"

Stream names¶

Selecting a proper stream name is important, as TeskaLabs LogMan.io determines a technology, selects proper parsing rules, dashboards and other content, based on the name of the stream.

To connect a log source which exists in Library and automatically assign the correct event lane, stream name must match one of event lane templates located in /Templates/EventLanes/ folder in Library.

Below is a table outlining the stream names used by various technologies when connecting to the LogMan.io Collector. Replace the star "*" at the end of the stream name with arbitrary number. For example, you can use a counter (fortinet-fortigate-1, fortinet-fortigate-2, linux-rsyslog-1, ...) or port number (fortinet-fortigate-10000, fortinet-fortigate-20000, linux-rsyslog-30000, ...).

Technology name	Stream name
Bitdefender GravityZone	bitdefender-gravityzone-*
Broadcom Brocade Switch	broadcom-brocade-switch-*
Cisco ASA	cisco-asa-*
Cisco FTD	cisco-ftd-*
Cisco iOS	cisco-ios-*
Cisco ISE	cisco-ise-*
Cisco Switch Nexus	cisco-switch-nexus-*
Cisco WLC	cisco-wlc-*
Dell iDRAC	dell-idrac-*
Dell PowerVault	dell-powervault-*
Dell Switch	dell-switch-*
Devolutions Web Server	devolutions-web-server-*
EATON UPS	eaton-ups-*
ESET Protect	eset-protect-*
F5	f5-*
FileZilla	filezilla-*
Fortinet FortiClient	fortinet-forticlient-*
Fortinet FortiGate	fortinet-fortigate-*
Fortinet FortiMail	fortinet-fortimail-*
Fortinet FortiSwitch	fortinet-fortiswitch-*
Gordic Ginis	gordic-ginis-*
Helios	helios-*
HPE Aruba ClearPass	hpe-aruba-clearpass-*
HPE Aruba IAP	hpe-aruba-iap-*
HPE Aruba IAP	hpe-aruba-switch-*
HPE iLO	hpe-ilo-*
HPE LaserJet Series	hpe-laserjet-*
HPE Primera	hpe-primera-*
HPE StoreOnce	hpe-storeonce-*
IBM QRADAR	ibm-qradar-*
IceWarp	icewarp-mailserver-*
Kubernetes	kubernetes-*
Linux Auditd	linux-auditd-*
Linux Rsyslog	linux-rsyslog-*
Linux Syslog RFC 3164	linux-syslog-rfc3164-*
Linux Syslog RFC 5424	linux-syslog-rfc5424-*
McAfee Webwasher	mcafee-webwasher-*
MikroTik	mikrotik-*
Minolta Bizhub	minolta-bizhub-*
Nginx	nginx-*
Ntopng	ntopng-*
OpenVPN	openvpn-*
Oracle Cloud	oracle-cloud-*
Oracle Listener	oracle-listener-*
Oracle Spark	oracle-spark-*
PfSense	pfsense-*
SentinelONE	sentinelone-*
Sophos Device Standard Format	sophos-device-standard-format-*
Sophos Standard Syslog Protocol	sophos-standard-syslog-protocol-*
Sophos Unstructured Format	sophos-unstructured-*
Squid Proxy	squid-proxy-*
Synology NAS	synology-nas-*
Ubiquiti UniFi	ubiquiti-unifi-*
Veeam Backup & Replication	veeam-backup-replication-*
VMware Cloud Director	vmware-cloud-director-*
VMware ESXi	vmware-esxi-*
VMware vCenter	vmware-vcenter-*
ySoft SafeQ	ysoft-safeq-*
ZyXEL Switch	zyxel-switch-*

Example

Configuration example of three log sources connected over TCP/UDP:

Two sources of Fortinet FortiGate on ports 10000 and 20000
One source of Linux Rsyslog on port 30000

lmio-collector.yaml

# Fortinet FortiGate on port 10000
input:TCP:fortinet-fortigate-1:
  address: 10000
  output: fortinet-fortigate-1

output:CommLink:fortinet-fortigate-1: {}

# Fortinet FortiGate on port 20000
input:TCP:fortinet-fortigate-2:
address: 20000
output: fortinet-fortigate-2

output:CommLink:fortinet-fortigate-2: {}

# Linux Rsyslog on port 30000
input:TCP:linux-rsyslog-1:
address: 30000
output: linux-rsyslog-1

output:CommLink:linux-rsyslog-1: {}