Collecting logs from Syslog¶

To collect logs from various Syslog sources, TeskaLabs LogMan.io Collector provides an option to setup a TCP and/or UDP receival port that conforms to a wide variety of Syslog protocols. You can simply configure a log source (for example Linux server or a network appliance) to send logs to the collector, typically to the port 514 (TCP and/or UDP) and TeskaLabs LogMan.io Collector will ingest these logs.

Tip

Syslog protocol typically uses port 514, TCP or UDP, but TeskaLabs LogMan.io Collector can be configured to use other ports as well.

To utilize this feature, configure your log sources to forward logs to port 514, 1514 or 6514, UDP, TCP or SSL of the TeskaLabs LogMan.io Collector. TeskaLabs LogMan.io Collector will automatically detect the origin of the incoming data and categorize it into streams (event lanes) accordingly.

Smart classification¶

TeskaLabs LogMan.io Collector classifies incoming logs using a smart feature. This enables a convenient ability to configure all log sources to send logs to a TeskaLabs LogMan.io Collector IP address and a specific port, including a very simple network path configuration.

TeskaLabs LogMan.io Collector uses a classification map in its YAML configuration.

Configuration¶

This example shows a configuration of TeskaLabs LogMan.io Collector for using Syslog and a smart classification of the incoming logs on port 514 UDP and TCP.

Stream name is important

A stream name specified in configuration of LogMan.io Collector is important. LogMan.io selects a proper parsing rules, content and other components related to the log source based on the stream name. Read more about stream names here.

Minimal configurationConfiguration with classified streams

Here is an example of a minimal configuration in which all incoming logs will be sent to a stream generic.

classification:
  smart_syslog: &smart_syslog  # YAML anchor referencing to both SmartDatagram and SmartStream inputs

    # All events are sent to stream 'generic'
    generic:
      - ip: "*"

# Listen on the UDP 514
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart_syslog
  output: smart

# Listen on the TCP 514
input:SmartStream:TCP514:
  address: 514
  smart: *smart_syslog
  output: smart

# Listen on the SSL 6514
input:SmartStream:SSL6514:
  address: 6514
  ssl: on
  smart: *smart_syslog
  output: smart

# Connection to LogMan.io
output:CommLink:smart: {}

Here is an example of configuration with four streams linux-syslog-rfc5424-1, fortinet-fortigate-1, fortinet-fortigate-2 and linux-rsyslog-1 with various options of how you can classify the incoming IP addresses, ports and protocols.

classification:

  smart514: &smart514  # YAML anchor referencing to both SmartDatagram and SmartStream inputs

    linux-syslog-rfc5424-1:    # stream name

      - ip: "192.168.0.1"      # Single IPv4 address
        port: 80               # Single port
        protocol: TCP          # TCP protocol

      - ip: "2001:db8::1"      # Single IPv6 address
        port: "1000-2000"      # Port range
        protocol: UDP          # UDP protocol

    fortinet-fortigate-1:

      - ip: "10.0.0.0/8"       # IPv4 range
        port: 14000
        protocol: UDP

      - ip: "fd00::/8"         # IPv6 range
        port: "*"              # Any port
        protocol: UDP

    fortinet-fortigate-2:

      - ip: "*"                # Any IP address
        port: "*"              # Any port
        protocol: UDP

    linux-rsyslog-1:
      - ip: "::1"              # Local IP addresses

# Listen on the UDP 514
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart514
  output: smart

# Listen on the TCP 514 with SSL auto-detection
input:SmartStream:TCP514:
  address: 514
  ssl: auto
  smart: *smart514
  output: smart

# The logs are forwarded to a LogMan.io using a CommLink
output:CommLink:smart: {}

Warning

Smart classification works only with a CommLink output.

SmartDatagram and SmartStream¶

SmartDatagram (for UDP) and SmartStream (for TCP) sources are similar to Datagram/TCP and Stream/UDP sources, with additional option smart, which references the appropriate sub-section in classification.

Listening on UDP¶

The configuration for listening on UDP port 514.

input:SmartDatagram:UDP514:
  address: 514
  smart: *smart_syslog
  output: smart

Listening on TCP¶

The configuration for listening on TCP port 514.

input:SmartStream:TCP514:
  address: 514
  smart: *smart_syslog
  output: smart

Listening on SSL/TLS¶

The TLS/SSL can be enabled on the input:SmartStream (TCP) by ssl configuration option.

input:SmartStream:SSL6514:
  address: 6514
  ssl: on
  key: key.pem
  cert: cert.pem
  smart: *smart_syslog
  output: smart

Possible values of ssl options are auto / on / off, the default is auto. auto means that both plain TCP and SSL/TLS is accepted, with the autodetection.

If the private key (key) and respective certificate (cert) is not provided, the internal Certification Authority of the collector will generate a self-signed SSL certificate.

Tip

More advanced TCP/SSL settings are described below.

Classification map¶

Section classification can contain one or more classifiers.

Warning

Section classification must be specified BEFORE input:... sections, otherwise the reference (i.e. *smart_syslog) is not recognized.

Each classifier specifies a combination of IP address ranges, port ranges and protocols; and it resolves them into a stream. Every log that arrives to the TeskaLabs LogMan.io Collector smart syslog is matched with these classifiers and a resulting stream is used as its destination in LogMan.io. If no match is found, the log goes into generic stream (named generic).

Tip

Streams can be found in the Archive component of TeskaLabs LogMan.io.

linux-syslog-rfc5424-1:
  - ip: "192.168.0.1"
    port: 80
    protocol: TCP

  - ip: "2001:db8::1"
    port: "1000-2000"
    protocol: UDP

linux-syslog-rfc5424-1 is a stream name.

ip¶

A single IPv4/IPv6 address: 92.168.0.1, 2001:db8::1
Range of IPv4/IPv6 addresses: 10.0.0.0/8, fd00::/8
Wildcard * for all IPv4/IPv6 addresses

port¶

A single port: 5400
Port range: 4000-8000
When not specified or *, range 0-65535 is used

protocol¶

TCP / UDP
When not specified, both TCP and UDP are used

* wildcard can go too wild

Be sure to wrap * wildcard into quotation marks in YAML "*". Aterisk without quotation marks would break YAML syntax.

Overlapping IP addresses and ports¶

IP addresses and ports can overlap. In that case, the most specific match is selected. In the example below, 25400 is matched with fortinet-fortigate-3, 25100 with fortinet-fortigate-2 and 24000 with fortinet-fortigate-1:

fortinet-fortigate-1:
  - ip: "192.168.0.1"
    port: 24000-30000

fortinet-fortigate-2:
  - ip: "192.168.0.1"
    port: 25000-26000

fortinet-fortigate-3:
  - ip: "192.168.0.1"
    port: 25400

The same holds for IP addresses.

Generic stream¶

If no target stream is identified during the classification, a log is forwarded to generic stream.

Example of stream classification

Suppose you connect new log sources from IP address range 192.168.0.0/24.

With no classifier, events are collected into generic stream and stored in Archive.

After looking into streams in Archive, you discovered there is a source of type logsox. You classify the stream to separate it from other incoming data:

logsox-1:
  - ip: "192.168.0.0/24"

When creating parsing rules for the stream, you discover that logsox incoming events from IP 192.168.0.68 have a different form than the others. You can isolate that stream and apply different parsing rules for it:

logsox-1:
  - ip: "192.168.0.0/24"

logsox-2:
  - ip: "192.168.0.68"

Stream names¶

Selecting a proper stream name is important, as TeskaLabs LogMan.io determines a technology, selects proper parsing rules, dashboards and other content, based on the name of the stream.

To connect a log source which exists in Library and automatically assign the correct event lane, stream name must match one of event lane templates located in /Templates/EventLanes/ folder in Library.

Below is a table outlining the stream names used by various technologies when connecting to the LogMan.io Collector. Replace the star "*" at the end of the stream name with arbitrary number. For example, you can use a counter (fortinet-fortigate-1, fortinet-fortigate-2, linux-rsyslog-1, ...) or port number (fortinet-fortigate-10000, fortinet-fortigate-20000, linux-rsyslog-30000, ...).

Technology name	Stream name
Apache HTTP Server	apache-http-server-*
Bitdefender Cloud Security	bitdefender-cloud-*
Bitdefender GravityZone	bitdefender-gravityzone-*
Blue Coat Secure Web Gateway	broadcom-blue-coat-swg-*
Broadcom Brocade Switch	broadcom-brocade-switch-*
CEF	cef-*
Check Point Firewall	check-point-firewall-*
Cisco ACI	cisco-aci-*
Cisco ASA	cisco-asa-*
Cisco FTD	cisco-ftd-*
Cisco IOS	cisco-ios-*
Cisco ISE	cisco-ise-*
Cisco MDS	cisco-mds-*
Cisco Switch Catalyst	cisco-switch-catalyst-*
Cisco Switch Nexus	cisco-switch-nexus-*
Cisco UCS	cisco-ucs-*
Cisco WLC	cisco-wlc-*
Citrix	citrix-*
Dell iDRAC	dell-idrac-*
Dell PowerVault	dell-powervault-*
Dell Switch	dell-switch-*
Devolutions Web Server	devolutions-web-server-*
EATON UPS	eaton-ups-*
ESET Protect	eset-protect-*
F5	f5-*
FileZilla	filezilla-*
Fortinet FortiClient	fortinet-forticlient-*
Fortinet FortiGate	fortinet-fortigate-*
Fortinet FortiMail	fortinet-fortimail-*
Fortinet FortiSwitch	fortinet-fortiswitch-*
Generic	generic
Gordic Ginis	gordic-ginis-*
HAProxy	haproxy-*
Helios	helios-*
HPE Aruba ClearPass	hpe-aruba-clearpass-*
HPE Aruba IAP	hpe-aruba-iap-*
HPE Aruba IAP	hpe-aruba-switch-*
HPE iLO	hpe-ilo-*
HPE LaserJet Series	hpe-laserjet-*
HPE Primera	hpe-primera-*
HPE StoreOnce	hpe-storeonce-*
IBM QRADAR	ibm-qradar-*
IBM Tape Library	ibm-tape-library-*
IceWarp	icewarp-mailserver-*
Ivanti Security	ivanti-security-*
Kubernetes	kubernetes-*
Lenovo XClarity Controller	lenovo-xcc-*
Linux Auditd	linux-auditd-*
Linux Rsyslog	linux-rsyslog-*
Linux Syslog RFC 3164	linux-syslog-rfc3164-*
Linux Syslog RFC 5424	linux-syslog-rfc5424-*
ManageEngine AD Audit Plus	manageengine-ad-audit-plus-*
McAfee Webwasher	mcafee-webwasher-*
Microsoft DHCP	microsoft-dhcp-*
Microsoft DNS	microsoft-dns-*
Microsoft Exchange	microsoft-exchange-*
Microsoft IIS	microsoft-iis-*
Microsoft SQL Server	microsoft-sql-server-*
MikroTik	mikrotik-*
Minolta Bizhub	minolta-bizhub-*
NetApp FAS	netapp-fas-*
NetApp Storage	netapp-storage-*
Nginx	nginx-*
Ntopng	ntopng-*
OpenStack Audit HTTP Request	openstack-*
OpenVPN	openvpn-*
Oracle Cloud	oracle-cloud-*
Oracle Listener	oracle-listener-*
Oracle Spark	oracle-spark-*
PaloAlto	palo-alto-*
PfSense	pfsense-*
QNAP NAS	qnap-nas-*
Samba Active Directory Domain Controller	samba-ad-dc-*
SentinelONE	sentinelone-*
Siemens Scalance	siemens-scalance-*
Sophos Device Standard Format	sophos-device-standard-format-*
Sophos Standard Syslog Protocol	sophos-standard-syslog-protocol-*
Sophos Unstructured Format	sophos-unstructured-*
Squid Proxy	squid-proxy-*
Synology NAS	synology-nas-*
Syslog RFC 3164	syslog-rfc3164-*
Syslog RFC 5424	syslog-rfc5424-*
Ubiquiti UniFi	ubiquiti-unifi-*
Veeam Backup & Replication	veeam-backup-replication-*
VMware Cloud Director	vmware-cloud-director-*
VMware ESXi	vmware-esxi-*
VMware vCenter	vmware-vcenter-*
Whalebone	whalebone-*
Microsoft Windows Events (via Winlogbeat)	winlogbeat
Wowza Streaming Engine	wowza-*
ySoft SafeQ	ysoft-safeq-*
ZyXEL Firewall	zyxel-firewall-*
ZyXEL Switch	zyxel-switch-*

Example

Configuration example of three log sources connected over TCP/UDP:

Two sources of Fortinet FortiGate on ports 10000 and 20000
One source of Linux Rsyslog on port 30000

lmio-collector.yaml

# Fortinet FortiGate on port 10000
input:TCP:fortinet-fortigate-1:
  address: 10000
  output: fortinet-fortigate-1

output:CommLink:fortinet-fortigate-1: {}

# Fortinet FortiGate on port 20000
input:TCP:fortinet-fortigate-2:
  address: 20000
  output: fortinet-fortigate-2

output:CommLink:fortinet-fortigate-2: {}

# Linux Rsyslog on port 30000
input:TCP:linux-rsyslog-1:
  address: 30000
  output: linux-rsyslog-1

output:CommLink:linux-rsyslog-1: {}

Dedicated syslog ports¶

TeskaLabs LogMan.io Collector can be also configured so that it provides a dedicated port(s) for a specific Syslog / TCP / UDP / SSL log source. This is useful in a more complex setups.

The Syslog protocol comes in (at least) two variants: the older BSD syslog / RFC3164 and a more recent IETF / RFC5424.

TCP vs UDP

TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are both protocols used for sending data over the network.

TCP is a Stream, as it provides a reliable, ordered, and error-checked delivery of a stream of data.

In contrast, UDP is a datagram that sends packets independently, allowing faster transmission but with less reliability and no guarantee of order, much like individual, unrelated messages.

Logs should be shipped through TCP protocol. Only if it is not possible, use UDP protocol.

Tip

For troubleshooting, use tcpdump to capture raw network traffic and then use Wireshark for deeper analysis.

The example of capturing the traffic at TCP/10008 port:

$ sudo tcpdump -i any tcp port 10008 -s 0 -w /tmp/capute.pcap -v

When enough traffic is captured, press Ctrl-C and collect the file /tmp/capture.pcap that contains the traffic capture. This file can be opened in Wireshark.

Listening on dedicated TCP¶

input:TCP:syslogtcp:
  address: 514
  output: syslog

Alternative section name

input:TCP has an equivalent alternative input:Stream since it can be used to consume logs on any stream socket such as UNIX socket.

Listening on dedicated UDP¶

input:UDP:syslogudp:
  address: 514
  output: syslog

Alternative section name

input:UDP has an equivalent alternative input:Datagram since it can be used to consume logs on any stream socket such as UNIX socket.

Other options are:

max_packet_size: Specify the maximum size of packets in bytes (default: 65536)
receiver_buffer_size: Limit the receiver size of the buffer in bytes (default: 0)

address¶

Here are possible form of address:

8080 or *:8080: Listen on a port 8080 all available network interfaces on IPv4 and IPv6
0.0.0.0:8080: Listen on a port 8080 all available network interfaces on IPv4
:::8080: Listen on a port 8080 all available network interfaces on IPv6
1.2.3.4:8080: Listen on a port 8080 and specific network interface (1.2.3.4) on IPv4
::1:8080: Listen on a port 8080 and specific network interface (::1) on IPv6
/tmp/unix.sock: Listen on a UNIX socket /tmp/unix.sock

Special cases of Syslog¶

Special cases of TCP input for parsing SysLog via TCP. For more information, see RFC 6587 and RFC 3164, section 4.1.1.

input:TCPBSDSyslogRFC6587:syslogrfc6587:
  address: 514
  output: syslog

Optional configuration:

max_sane_msg_len: Maximum size in bytes of SysLog message to be received (default: 10000)

input:TCPBSDSyslogNoFraming:syslognoframes:
  address: 514
  output: syslog

Optional configuration:

buffer_size: Maximum size in bytes of SysLog message to be received (default: 64 * 1024)
variant: The variant of SysLog format of the incoming message, can be auto, nopri with no PRI number in the beginning and standard with PRI (default: auto).

TLS/SSL¶

Any TCP input can be configured to apply TLS / SSL. It is done by specifying key and cert options and/or ssl parameter for smart syslog inputs.

Example of TLS/SSL that requires a client to present SSL certificate issued by trusted CA.

input:TCP:ssl:
  address: 6514
  output: syslog

  cert: /conf/server_cert_chain.pem
  key: /conf/server_key.pem

  verify_mode: CERT_REQUIRED
  cafile: ./conf/ca.pem

The following configuration options specify the TLS / SSL connection:

cert: Path to the server SSL certificate (or certificate chain) in PEM format
key: Path to the private key of the client SSL certificate in PEM format
key_password: Private key file password (optional, default: none)
cafile: Path to a PEM file with CA certificate(s) to verify the SSL client (optional, default: none)
capath: Path to a directory with CA certificate(s) to verify the SSL client (optional, default: none)
cadata: one or more PEM-encoded CA certificates to verify the SSL client (optional, default: none)
verify_mode: One of CERT_NONE, CERT_OPTIONAL or CERT_REQUIRED (optional)
ciphers: SSL ciphers (optional, default: none)
dh_params: Diffie–Hellman (D-H) key exchange (TLS) parameters (optional, default: none)

Tip

When the collector is operated in the container, the path prefix for PEM files is /conf/.
The /conf/ directory within the container is mapped to a respective configuration directory on the host.