Skip to content

Collecting logs with Elastic Agents via Fleet

This guide describes how to deploy managed Elastic Agents through Fleet Server and forward their output to LogMan.io Collector using the Logstash output. Agents are centrally managed in Kibana Fleet while log data is processed in the LogMan.io pipeline.

Prerequisites

Before you start, ensure you have:

  • LogMan.io Collector with Beats input enabled
  • LogMan.io v26.12 or higher
  • Elasticsearch and Kibana versions 8.19 or higher (compatible with the Elastic Agent you deploy)
  • Access to the certificate authority (CA) used by LogMan.io Collector
  • A server where Fleet Server will run

1. Prepare certificates for the Collector Logstash input

Secure communication between Elastic Agents (or Fleet Server) and LogMan.io Collector requires a server certificate on the Collector and a client certificate for the Fleet Logstash output.

The Collector usually already has a certificate. On integrated Collectors, it may be issued for a different IP address than the one sources connect to, so you may need to reissue it.

1.1 Update the Collector server certificate

Generate a new CSR with the Collector ingress IP address:

openssl req -new \
  -key ssl-server-key.pem \
  -out server-new.csr \
  -subj "/O=TeskaLabs/OU=LogMan.io Collector/CN={INGRESS_IP_ADDRESS}"

Create an extension file for the certificate:

cat > server_ext.cnf << 'EOF'
[ v3_req ]
basicConstraints = critical, CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, keyAgreement
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
subjectAltName = DNS:10.44.170.82
EOF

Adjust subjectAltName to match your Collector address.

Sign the certificate with the CA:

openssl x509 -req \
  -in server-new.csr \
  -CA ca-cert.pem \
  -CAkey ca-key.pem \
  -CAcreateserial \
  -out server-new.crt \
  -days 3650 \
  -sha256 \
  -extfile server_ext.cnf \
  -extensions v3_req

Back up and replace the existing certificate:

cp ssl-server-cert.pem ssl-server-cert.pem.bak
cp ssl-server-key.pem ssl-server-key.pem.bak
cp server-new.crt ssl-server-cert.pem

Restart the Collector to load the new certificate.

1.2 Create a client certificate for Logstash output

Generate a client key and certificate for the Fleet output configuration:

openssl genpkey -algorithm EC \
  -pkeyopt ec_paramgen_curve:P-256 \
  -out logstash-client-key.pem

openssl req -new \
  -key logstash-client-key.pem \
  -out logstash-client.csr \
  -subj "/O=TeskaLabs Ltd/OU=LogMan.io Collector/CN=Logstash Client"

openssl x509 -req \
  -in logstash-client.csr \
  -CA ca-cert.pem \
  -CAkey ca-key.pem \
  -CAcreateserial \
  -out logstash-client-cert.pem \
  -days 3650 \
  -extfile <(printf "keyUsage=digitalSignature\nextendedKeyUsage=clientAuth\n")

Files needed in Fleet Settings

You will need logstash-client-cert.pem, logstash-client-key.pem, and ca-cert.pem when configuring the Logstash output in Fleet Settings (step 5).

2. Initialize Fleet Server

Fleet Server is the central component through which Kibana manages Elastic Agents. Initialize it in Kibana:

  1. Go to Kibana → Fleet → Settings.
  2. Click Add a Fleet Server.
  3. In the dialog, set:
    • Name: Fleet Server name (for example logman-node-1)
    • URL: public IP address or hostname with port 8220 (for example https://10.20.30.40:8220)
  4. Click Continue.

After the Fleet Server policy is created, Kibana shows installation commands. On the target Linux server, run:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.19.12-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.19.12-linux-x86_64.tar.gz
cd elastic-agent-8.19.12-linux-x86_64

sudo ./elastic-agent install \
  --fleet-server-es=http://localhost:9200 \
  --fleet-server-service-token=<GENERATED_TOKEN> \
  --fleet-server-policy=<POLICY_NAME> \
  --fleet-server-port=8220

Installation command

Kibana generates the exact command, including the service token and policy name. Values differ for each installation.

After a successful install, Fleet confirms that Fleet Server is connected.

3. Configure Fleet Server policy

After Fleet Server is initialized, adjust the default Fleet Server policy so the Fleet Server agent does not send its own logs and metrics directly to Elasticsearch:

  1. Go to Kibana → Fleet → Agent policies → Fleet Server Policy → Settings.
  2. Under Agent monitoring, disable:
    • Collect agent logs
    • Collect agent metrics

Fleet Server then acts purely as a management component.

4. Create a Logstash output

Redirect data from Elastic Agents to LogMan.io Collector using the Logstash protocol:

  1. Go to Kibana → Fleet → Settings.
  2. Under Outputs, click Add output.
  3. Set:
    • Name: logstash-collector (or another descriptive name)
    • Type: Logstash
    • Logstash hosts: LogMan.io Collector IP address and port (for example 10.20.30.40:5044)
    • Server SSL certificate authorities: contents of ca-cert.pem
    • Client SSL certificate: contents of logstash-client-cert.pem
    • Client SSL certificate key: contents of logstash-client-key.pem
  4. Enable Make this output the default for agent integrations.
  5. Click Save and apply settings.

The new Logstash output should appear in the Outputs list with Agent integrations marked as default.

5. Create agent policies for log collection

Create a separate agent policy for each collection type or agent group:

  1. Go to Kibana → Fleet → Agent policies.
  2. Click Create agent policy.
  3. Name the policy and set:
    • Output for integrations: logstash-collector
    • Fleet Server: your Fleet Server (for example logman-node-1)
  4. Add the required integrations (for example System, Windows, Custom logs).

Verify output on every policy

For each new policy, confirm that Output for integrations is set to logstash-collector, not the default Elasticsearch output. Otherwise logs bypass the LogMan.io pipeline.

6. Install Elastic Agents

Install Elastic Agents on target systems and enroll them into agent policies:

  1. Go to Kibana → Fleet → Agents.
  2. Click Add agent.
  3. Select the agent policy.
  4. Follow the installation instructions for the target operating system (Linux, Windows, macOS).

After successful installation and enrollment, the agent appears in Fleet and starts sending data through the Logstash output to LogMan.io Collector.

Architecture

graph LR
  agent[Elastic Agent<br/>target host]
  fleet[Fleet Server]
  collector[LogMan.io Collector<br/>Beats input]
  pipeline[LogMan.io<br/>pipeline]
  kibana[Kibana<br/>Fleet]

  agent --> fleet
  fleet --> collector
  collector --> pipeline
  kibana --> fleet
  kibana --> agent

Elastic Agents are centrally managed through Kibana Fleet. Log data flows over the Logstash protocol directly to LogMan.io Collector and then through the standard LogMan.io processing pipeline.

Troubleshooting

  • Fleet Server does not connect: Verify port 8220 is reachable and the service token is correct.
  • Agent does not send data: Check that the agent policy uses output logstash-collector and that the Collector is reachable at the configured address and port.
  • Certificate errors: Confirm the client certificate was signed by the same CA as the Collector server certificate, and that SAN matches the address agents connect to.
  • Logs appear in Elasticsearch instead of LogMan.io: In Fleet Settings, ensure logstash-collector is the default output for agent integrations.