Collecting from Beats or Logstash¶

Supported Beats¶

TeskaLabs LogMan.io is designed to collect logs from Elastic Beats, a suite of lightweight data shippers that extract and forward log data from diverse sources and platforms. The Collector supports all official Beats distributions. The following are among the most commonly utilized:

Winlogbeat: Collects Windows Event logs from a wide range of channels, including Application, System, Security, and specialized event sources. This Beat is essential for monitoring Windows servers and workstations.
Filebeat: Collects and forwards logs from files in real-time. This Beat is utilized for a variety of logging scenarios, including:
- Microsoft DHCP: Captures DHCP server logs from Windows servers for network diagnostics and auditing.
- Microsoft DNS: Captures DNS debug logs from Windows DNS servers for DNS query analysis.
- Microsoft IIS: Captures web server logs from Microsoft Internet Information Services.
- Microsoft Exchange: Captures email server logs from Microsoft Exchange installations.
- Oracle Database: Captures audit logs from Oracle database systems for compliance and security monitoring.
- Custom application logs: Captures logs from any custom application or service that writes to file-based outputs.
Auditbeat: Captures system audit logs from Linux environments, including file integrity monitoring and user activity tracking.
Other Beats: Additional official and community-maintained Beats (such as Metricbeat and Packetbeat) are also supported through the same integration framework.

Data Transport Protocol¶

Beats transmit log data to the LogMan.io Collector using the Lumberjack protocol, which is the same protocol employed by Logstash. This protocol ensures secure, reliable, and efficient log forwarding across diverse sources and platforms.

Integration Requirements¶

To enable log collection from Beats, you must:

Configure the LogMan.io Collector to accept incoming Beats connections on the designated port (default: TCP port 5044)
Configure each Beats instance to forward logs to the Collector's network address and configured port

Once configured, the Collector will receive the incoming logs and route them to their configured destinations according to your deployment specifications.

Continue to the LogMan.io Collector configuration for Beats.
Continue to the Beats configuration.

Configuring the LogMan.io Collector¶

Basic Configuration¶

The minimal configuration of LogMan.io Collector is as follows:

input:Beats:beats:
  output: beats

output:CommLink:beats: {}

By default, the Beats input listens for incoming connections on TCP port 5044.

Stream Naming Convention

The output: beats designation differs from other standard Collector outputs in that it does not define a static stream name. Instead, the stream name is dynamically determined by the stream field present in each incoming log event from the Beats instance. This approach provides enhanced flexibility in log routing and organization, enabling logs to be categorized and processed according to their source or type as defined within the Beats configuration itself.

The following example shows how to set the stream field in a Filebeat configuration:

filebeat.yaml

filebeat.inputs:
  - type: filestream
    fields:
      stream: application-1

Protocol Details

The Lumberjack protocol uses TCP port 5044 by default and supports optional SSL/TLS encryption for secure log transmission. From an operational perspective, the input source classes input:Lumberjack:, input:Logstash:, and input:Beats: are functionally equivalent and can be used interchangeably, as they all implement the same underlying protocol.

Network Address Configuration¶

The optional address attribute specifies the network port on which the Collector listens for incoming Beats connections. If not specified, the Collector defaults to port 5044.

input:Beats:beats:
  output: beats
  address: 5044

output:CommLink:beats: {}

SSL/TLS Configuration¶

Incoming SSL/TLS connections are automatically detected and established by the Collector. The Beats input supports comprehensive SSL/TLS configuration through the following parameters:

cert: Path to the client SSL certificate
key: Path to the private key of the client SSL certificate
password: Private key file password (optional, default: none)
cafile: Path to a PEM file with CA certificate(s) to verify the SSL server (optional, default: none)
capath: Path to a directory with CA certificate(s) to verify the SSL server (optional, default: none)
cadata: one or more PEM-encoded CA certificates to verify the SSL server (optional, default: none)
ciphers: SSL ciphers (optional, default: none)
dh_params: Diffie–Hellman (D-H) key exchange (TLS) parameters (optional, default: none)
verify_mode: One of CERT_NONE, CERT_OPTIONAL or CERT_REQUIRED (optional); for more information, see: github.com/TeskaLabs/asab

Smart Map Integration¶

The optional smart attribute allows you to specify a smart map for enhanced log routing and processing when the Beats input is connected to a output:CommLink destination. Smart maps enable dynamic, rule-based log categorization and destination selection.

input:Beats:beats:
  output: beats
  smart: my-smart-map

output:CommLink:beats: {}

Configuring Beats¶

This section describes the configuration requirements for Beats instances to forward logs to the LogMan.io Collector. Beats are lightweight data shippers provided by Elastic that extract, process, and forward logs to centralized collection points.

The following configuration snippet shows the basic output configuration required for all Beats to forward logs to the LogMan.io Collector:

output.logstash:
  hosts: ["<collector>:5044"]

Replace <collector> with the network address or hostname of your LogMan.io Collector instance.

Winlogbeat Configuration¶

Winlogbeat is configured to collect Windows Event logs. The following example demonstrates a typical Winlogbeat configuration for log collection from multiple event channels:

winlogbeat.yaml

output.logstash:
  hosts: ["<collector>:5044"]

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h

  - name: System
    ignore_older: 72h

  - name: Security
    ignore_older: 72h

  - name: Setup
    ignore_older: 72h

  - name: Microsoft-Windows-Sysmon/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-Windows Defender/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-GroupPolicy/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-TaskScheduler/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall
    ignore_older: 72h

  - name: Windows PowerShell
    event_id: 400, 403, 600, 800
    ignore_older: 72h

  - name: Microsoft-Windows-PowerShell/Operational
    event_id: 4103, 4104, 4105, 4106
    ignore_older: 72h

  - name: ForwardedEvents
    tags: [forwarded]

logging.to_files: true
logging.files:
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 7

Windows Security Event Log Access¶

For Winlogbeat to successfully collect Windows Security event logs, the user account under which the Winlogbeat service operates must be granted the appropriate permissions. Specifically, this user account must be a member of the Event Log Readers group on the local system.

Procedure: Adding Winlogbeat to Event Log Readers Group¶

The following procedure demonstrates how to add the Winlogbeat service account to the Event Log Readers group using the Local Users and Groups manager:

Open the Run dialog:
Press Win + R on your keyboard.
Type lusrmgr.msc and press Enter.
Navigate to Groups:
In the left pane, click on Groups.
Find and open the group:
In the right pane, double-click on Event Log Readers.
Add a new member:
In the window that appears, click on the Add... button.
Select the user or service:
In the Enter the object names to select field, type the name of the user or service account you want to add.
- For a service account like Winlogbeat, you might use:
```
NT SERVICE\Winlogbeat
```
* For a regular local user, simply type the username (e.g. winlogbeat_user).
Check and confirm:
Click Check Names to validate the input.
Click OK to confirm.
Apply and close:
Click Apply to confirm the group membership change, then click OK to close all windows.
Restart the Winlogbeat service:
Open PowerShell or Command Prompt with administrator privileges.
Execute the following command to apply the group membership changes:
```
Restart-Service winlogbeat
```

Filebeat Configuration¶

Filebeat is configured to collect logs from local files and forward them to the Collector. The following example demonstrates the basic configuration structure:

filebeat.yaml

output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  - type: filestream
    fields:
      stream: <stream name>
    paths:
      - /path/to/the/file.log

Microsoft DHCP¶

This configuration enables Filebeat to collect and forward DHCP server logs from Windows systems. The example below covers both IPv4 and IPv6 DHCP logs:

filebeat.yaml

output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:

  # Microsoft DHCP IPv4
  - type: filestream
    id: microsoft-dhcp-ipv4
    fields:
      stream: microsoft-dhcp-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1780
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\DHCP\DhcpSrvLog-*.log
    include_lines:
      - "^[0-9]+,"

  # Microsoft DHCP IPv6
  - type: filestream
    id: microsoft-dhcp-ipv6
    fields:
      stream: microsoft-dhcp-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1238
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\DHCP\DhcpV6SrvLog-*.log
    include_lines:
      - "^[0-9]+,"

Microsoft DNS¶

This configuration enables Filebeat to collect DNS debug logs from Windows DNS servers. The fingerprint settings ensure accurate log tracking and duplicate prevention:

filebeat.yaml

output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  # Microsoft DNS Debug Log
  - type: filestream
    id: microsoft-dns
    fields:
      stream: microsoft-dns-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1052
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\dns\debug.log
    include_lines:
      - "^[0-9]+[/-:][0-9]+"

Microsoft IIS¶

This configuration enables Filebeat to collect web server logs from Microsoft Internet Information Services:

filebeat.yaml

output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  # Microsoft IIS log
  - type: filestream
    id: microsoft-iis
    fields:
      stream: microsoft-iis-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1052
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\inetpub\logs\LogFiles\W3SVC*\*.log

Microsoft SQL Server¶

This configuration enables Filebeat to collect SQL Server error logs. Note that UTF-16 encoding is required for proper log parsing:

filebeat.yaml

output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:

  - type: filestream
    id: microsoft-sql-filebeat-v1
    enabled: true
    fields:
      stream: microsoft-sql-filebeat-v1
    paths:
      - C:\Program Files\Microsoft SQL Server\MSSQLXXXX.MSSQLSERVER\MSSQL\Log\ERRORLOG  # Replace with your MSSQL instance path
    encoding: utf-16

Oracle Database¶

This configuration enables Filebeat to collect Oracle database audit logs, which may be stored in XML or binary audit trail formats:

filebeat.yaml

output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  - type: log
    id: oracle-database-filebeat-v1
    enabled: true
    paths:
      - /var/oracle/audit/*.xml
      - /var/oracle/audit/*.aud
    fields:
      log_type: oracle_audit
      environment: production
      stream: oracle-database-filebeat-v1
    multiline.pattern: '^<Audit'
    multiline.negate: true
    multiline.match: after