Skip to content

Collecting from Beats or Logstash

Configuration of TeskaLabs LogMan.io

The basic configuration snippet of the collector:

input:Lumberjack:logstash:
  output: ...

logstash is an input identified and can be replaced by anything unique within the configuration file

address specifies an network interface and/or a port that will be used for listening. The default value is '5044', to listen on tcp/5044.

Optional smart attribute can be used to specify a smart map, when connected to the output:CommLink.

Tip

The protocol used for this extraction is called Lumberjack and by default runs on tcp/5044 with optional SSL. Log source classes input:Lumberjack:, input:Logstash: and input:Beats: are actually synonyms.

SSL configuration

The incoming SSL is detected automatically.

The following configuration options specify the SSL connection:

  • cert: Path to the client SSL certificate
  • key: Path to the private key of the client SSL certificate
  • password: Private key file password (optional, default: none)
  • cafile: Path to a PEM file with CA certificate(s) to verify the SSL server (optional, default: none)
  • capath: Path to a directory with CA certificate(s) to verify the SSL server (optional, default: none)
  • cadata: one or more PEM-encoded CA certificates to verify the SSL server (optional, default: none)
  • ciphers: SSL ciphers (optional, default: none)
  • dh_params: Diffie–Hellman (D-H) key exchange (TLS) parameters (optional, default: none)
  • verify_mode: One of CERT_NONE, CERT_OPTIONAL or CERT_REQUIRED (optional); for more information, see: github.com/TeskaLabs/asab

Configuration of Beats

This log source can be used to collect logs using Beats family of tools. Beats are lightweight data shippers from Elastic.

Supported Beats:

Configuration snippet:

output.logstash:
  hosts: ["<collector>:5044"]

Example of winlogbeat configuration

winlogbeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h

  - name: System
    ignore_older: 72h

  - name: Security
    ignore_older: 72h

  - name: Setup
    ignore_older: 72h

  - name: Microsoft-Windows-Sysmon/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-Windows Defender/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-GroupPolicy/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-TaskScheduler/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall
    ignore_older: 72h

  - name: Windows PowerShell
    event_id: 400, 403, 600, 800
    ignore_older: 72h

  - name: Microsoft-Windows-PowerShell/Operational
    event_id: 4103, 4104, 4105, 4106
    ignore_older: 72h

  - name: ForwardedEvents
    tags: [forwarded]

logging.to_files: true
logging.files:
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 7

Security logs

Make sure the user, under which the Winlogbeat service runs, has a permission to access Security logs.

To do so, the user has to be part of the Event Log Readers group.

Step-by-Step (via lusrmgr.msc)

  1. Open the Run dialog:

  2. Press Win + R on your keyboard.

  3. Type lusrmgr.msc and press Enter.

  4. Navigate to Groups:

  5. In the left pane, click on Groups.

  6. Find and open the group:

  7. In the right pane, double-click on Event Log Readers.

  8. Add a new member:

  9. In the window that appears, click on the Add... button.

  10. Select the user or service:

  11. In the Enter the object names to select field, type the name of the user or service account you want to add.

    • For a service account like Winlogbeat, you might use:

    NT SERVICE\Winlogbeat
    * For a regular local user, simply type the username (e.g. winlogbeat_user).

  12. Check and confirm:

  13. Click Check Names to validate the input.

  14. Click OK to confirm.

  15. Apply and close:

  16. Click Apply and then OK to close all windows.

  17. Restart the Winlogbeat service:

  18. Open PowerShell or Command Prompt as administrator.

  19. Run:

    Restart-Service winlogbeat

Example of filebeat configuration

filebeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  - type: filestream
    fields:
      stream: <stream name>
    paths:
      - /path/to/the/file.log

Tip

<stream name> is for example my-app-log. You don't need to include tenant or any other prefix.

Example of filebeat configuration for Microsoft DHCP

filebeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:

  # Microsoft DHCP IPv4
  - type: filestream
    id: microsoft-dhcp-ipv4
    fields:
      stream: microsoft-dhcp-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1780
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\DHCP\DhcpSrvLog-*.log
    include_lines:
      - "^[0-9]+,"

  # Microsoft DHCP IPv6
  - type: filestream
    id: microsoft-dhcp-ipv6
    fields:
      stream: microsoft-dhcp-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1238
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\DHCP\DhcpV6SrvLog-*.log
    include_lines:
      - "^[0-9]+,"

Example of filebeat configuration for Microsoft DNS

filebeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  # Microsoft DNS Debug Log
  - type: filestream
    id: microsoft-dns
    fields:
      stream: microsoft-dns-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1052
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\dns\debug.log
    include_lines:
      - "^[0-9]+[/-:][0-9]+"