Forwarding Logs from Collector¶

The LogMan.io Collector can forward received logs to other systems, such as another collector instance, a SIEM system, or any other log management tool that accepts TCP or UDP connections. This allows you to create log forwarding pipelines where logs are received, optionally processed, and then forwarded to multiple destinations.

Overview¶

Log forwarding enables you to:

Replicate logs to multiple destinations for redundancy or compliance
Forward logs to other collectors in a distributed setup
Send logs to external SIEM systems or log management tools
Create log pipelines where logs flow through multiple processing stages

The forwarding mechanism works by adding a forward: component to your pipeline configuration. The forward component receives events from the same output as your input, allowing you to both process logs locally and forward them to remote destinations.

Basic Forwarding Configuration¶

TCP Forwarding¶

To forward logs over TCP to another system:

# Input receiving logs
input:TCP:TestTCPInput:
  address: 0.0.0.0:9999
  output: print1

# Forward logs to another system via TCP
forward:TCP:TestTCPForward:
  address: 127.0.0.1:9998
  output: print1

# Local output (optional)
output:Print:print1: {}

In this example:

Input (input:TCP:TestTCPInput) receives logs on port 9999 from any network interface (0.0.0.0)
Forward (forward:TCP:TestTCPForward) forwards the same logs to 127.0.0.1:9998 via TCP
Output (output:Print:print1) also processes the logs locally (prints them to console)

Both the forward and output components reference the same output ID (print1), which means they both receive events from the input.

UDP Forwarding¶

To forward logs over UDP:

# Input receiving logs
input:UDP:TestUDPInput:
  address: 0.0.0.0:9999
  output: print2

# Forward logs to another system via UDP
forward:UDP:TestUDPForward:
  address: 127.0.0.1:9998
  output: print2

# Local output (optional)
output:Print:print2: {}

The configuration is similar to TCP forwarding, but uses UDP protocol instead.

Forward Configuration Parameters¶

Address¶

The address parameter specifies the destination where logs should be forwarded:

forward:TCP:myforward:
  address: <IP_ADDRESS>:<PORT>
  output: <OUTPUT_ID>

IP_ADDRESS: The destination IP address or hostname (IPv4 or IPv6 supported)
PORT: The destination port number

Examples:

address: 192.168.1.100:514        # IPv4 address
address: [2001:db8::1]:514        # IPv6 address (use brackets)
address: collector.example.com:514 # Hostname

Output Reference¶

The output parameter must reference the same output ID used by the input:

input:TCP:myinput:
  output: shared_output  # Input uses this output

forward:TCP:myforward:
  output: shared_output  # Forward also uses the same output

output:Print:shared_output: {}  # Local output definition

This ensures that both the forward component and any local output receive the same events from the input.

Forwarding with Smart Classification¶

You can combine forwarding with smart classification to route logs based on source IP addresses, ports, and protocols. This is useful when you want to forward all classified logs to an external system while also processing them locally.

Example: Forwarding Classified Logs¶

This example shows how to use smart classification with forwarding for both UDP and TCP inputs:

classification:
  smart_syslog: &smart_syslog  # YAML anchor referencing to both SmartDatagram and SmartStream inputs

    # All events are sent to stream 'generic'
    generic:
      - ip: "*"

# Listen on the UDP 514
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart_syslog
  output: smart-udp

forward:UDP:SmartUDPForward:
  address: 127.0.0.1:9998
  output: smart-udp

# Connection to LogMan.io
output:CommLink:smart-udp: {}

# Listen on the TCP 514
input:SmartStream:TCP514:
  address: 514
  smart: *smart_syslog
  output: smart-tcp

forward:TCP:SmartTCPForward:
  address: 127.0.0.1:9998
  output: smart-tcp

# Connection to LogMan.io
output:CommLink:smart-tcp: {}

In this configuration:

Smart Classification: The smart_syslog classification map (defined with YAML anchor &smart_syslog) routes all incoming logs to the generic stream based on source IP.
UDP Input: Receives logs on UDP port 514, classifies them using the smart map, and sends them to output smart-udp.
UDP Forward: Forwards all UDP logs to 127.0.0.1:9998 while also sending them to LogMan.io.
TCP Input: Receives logs on TCP port 514, classifies them using the same smart map, and sends them to output smart-tcp.
TCP Forward: Forwards all TCP logs to 127.0.0.1:9998 while also sending them to LogMan.io.

Both UDP and TCP inputs use the same smart classification map (via YAML anchor), but have separate output IDs (smart-udp and smart-tcp) to allow independent forwarding configurations.

Example: Advanced Classification with Forwarding¶

For more complex scenarios where you want to classify logs into different streams and forward them:

classification:
  smart_forward: &smart_forward
    # Classify logs from specific IP ranges
    fortinet-fortigate-1:
      - ip: "192.168.1.0/24"
        port: "*"
        protocol: UDP

    linux-syslog-1:
      - ip: "10.0.0.0/8"
        port: "*"
        protocol: TCP

    # All other logs go to generic stream
    generic:
      - ip: "*"
        port: "*"
        protocol: "*"

# Input with smart classification
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart_forward
  output: smart_forward

# Forward all classified logs to external SIEM
forward:TCP:siem_forward:
  address: siem.example.com:514
  output: smart_forward

# Also send to local LogMan.io
output:CommLink:smart_forward:
  connection: commlink

Smart Classification and Forwarding

When using smart classification with forwarding, the forward component receives all logs from the smart classification output. The smart classification determines which stream logs belong to in LogMan.io, while the forward component sends all logs to the external destination. If you need to forward only specific log streams to different destinations, you can use separate inputs with different forward configurations, or use transform filters to create separate output paths.

Complete Example: Multi-Destination Forwarding¶

Here is a complete example that receives logs, processes them locally, and forwards them to multiple destinations:

# Connection to LogMan.io
connection:CommLink:commlink:
  url: https://recv.logman.example.com/

# Input receiving syslog
input:TCP:syslog_input:
  address: 0.0.0.0:514
  output: syslog_pipeline

# Forward to external SIEM system
forward:TCP:siem_forward:
  address: siem.company.com:514
  output: syslog_pipeline

# Forward to backup collector
forward:UDP:backup_collector:
  address: 192.168.1.200:1514
  output: syslog_pipeline

# Send to local LogMan.io
output:CommLink:syslog_pipeline:
  connection: commlink

In this configuration:

Logs are received on TCP port 514
Logs are forwarded to an external SIEM system via TCP
Logs are also forwarded to a backup collector via UDP
Logs are sent to the local LogMan.io instance via CommLink

All three destinations receive the same logs simultaneously.

Use Cases¶

1. Log Replication for Compliance¶

Forward logs to a compliance archive system while also processing them in LogMan.io:

input:TCP:compliance_input:
  address: 0.0.0.0:514
  output: compliance_pipeline

# Forward to compliance archive
forward:TCP:compliance_archive:
  address: archive.company.com:514
  output: compliance_pipeline

# Process in LogMan.io
output:CommLink:compliance_pipeline:
  connection: commlink

2. Distributed Collector Setup¶

Forward logs from edge collectors to a central collector:

# Edge collector configuration
input:UDP:edge_input:
  address: 0.0.0.0:514
  output: edge_pipeline

# Forward to central collector
forward:TCP:central_collector:
  address: central-collector.company.com:514
  output: edge_pipeline

# Optional: also process locally
output:CommLink:edge_pipeline:
  connection: commlink

3. SIEM Integration¶

Forward security logs to an external SIEM while maintaining local processing:

input:TCP:security_input:
  address: 0.0.0.0:1514
  output: security_pipeline

# Forward security events to SIEM
forward:TCP:siem_forward:
  address: siem.company.com:514
  output: security_pipeline

# Process in LogMan.io for correlation
output:CommLink:security_pipeline:
  connection: commlink

Troubleshooting¶

Verify Network Connectivity¶

Test if the collector can reach the forward destination:

# Test TCP connectivity
nc -v destination-ip destination-port

# Test UDP connectivity
nc -u -v destination-ip destination-port

Check Forward Configuration¶

Ensure that:

The output parameter in the forward component matches the output ID used by the input
The destination address and port are correct
Network firewall rules allow outbound connections to the destination
The destination system is listening on the specified port

Monitor Forward Status¶

Check collector logs for forward-related errors:

# View collector logs
docker logs -f lmio-collector

# Look for forward connection errors
docker logs lmio-collector | grep -i forward

Common Issues¶

Logs not reaching destination:
Verify network connectivity between collector and destination
Check firewall rules allow outbound connections
Ensure destination is listening on the correct port
Verify the forward address format is correct
Connection refused errors:
Destination may not be listening on the specified port
Firewall may be blocking the connection
Check destination system logs
Partial log forwarding:
UDP may drop packets if network is congested (consider TCP for reliability)
Check if destination system has sufficient capacity
Monitor collector resource usage

Best Practices¶

Use TCP for reliability: TCP ensures reliable delivery, while UDP is faster but may lose packets
Monitor forward destinations: Set up monitoring to ensure forward destinations are available
Use transforms for filtering: If you need to forward only specific logs, use transform filters
Test connectivity: Verify network connectivity before deploying forward configurations
Consider load balancing: For high-volume scenarios, consider forwarding to multiple destinations with load balancing