Skip to content

Upgrading Elasticsearch from 7.17.28 to 8.19.12

TeskaLabs LogMan.io v26.12 ships with Elasticsearch 7.17.28 by default and supports upgrading to 8.19.12. This guide describes a rolling upgrade: one instance at a time, tier by tier.

Important warnings

  • The upgrade is irreversible. After you move to Elasticsearch 8.19.12, you cannot roll back to a lower version.
  • All instances must run the same version of Elasticsearch. You cannot operate a cluster with mixed Elasticsearch versions. Every instance must eventually be upgraded to 8.19.12.

Prerequisites

  1. LogMan.io v26.12 or newer is required. Earlier LogMan.io releases cannot run with Elasticsearch 8.19.12.
  2. The default Elasticsearch version in LogMan.io v26.12 is 7.17.28. You override that default in model.yaml. Changing the version from the model is documented here.

Before you start

Check cluster health before you begin.

  • Log collection is not interrupted during the upgrade.
  • Logs are not written to Elasticsearch while the cluster status is red. Kafka acts as a buffer in the meantime.
  • Make sure Kafka does not have large lags before you start.
  • Confirm that the Elasticsearch cluster status is green.

Upgrade procedure

1. Restrict shard allocation

Just before the upgrade, limit shard allocation. In Kibana, open Dev Tools and run:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

2. Upgrade Elasticsearch instances, one at a time

Upgrade instances tier by tier. In model.yaml, find the elasticsearch section with the named instances. Pick the first cold instance and add version: 8.19.12:

services:
  elasticsearch:
   instances:
     cold-1:
       version: 8.19.12
       node: lmio-server-1

On the right side of the screen, select the node where this instance runs and click Apply.

3. Monitor each instance

You can verify the instance restart and the new version on Maintenance → Services. For additional metrics, use the Monitoring screen in Kibana. You will see that the node restarted and may not yet have all of its shards loaded. Wait until the cluster status turns yellow before continuing.

4. Continue through the remaining tiers

When the cluster is yellow, upgrade the next instance in the same tier (for example, cold-2) the same way. Continue with the remaining tiers in this order:

  1. cold
  2. warm
  3. hot
  4. master

5. Expect a monitoring gap on hot nodes

When you start upgrading hot nodes, monitoring data stored on them will be unavailable. Continue upgrading all tiers, including master nodes, even if you cannot monitor the cluster effectively during this phase. Monitoring becomes available again after the upgrade completes.

6. Upgrade Kibana

Upgrade Kibana to the matching version as well. An older Kibana release will not communicate with the new Elasticsearch version.

services:
  kibana:
   instances:
     1:
       version: 8.19.12
       node: lmio-server-1

7. Restore shard allocation

Once Kibana is running, monitoring is back, and the cluster is yellow, restore shard allocation in Kibana Dev Tools:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

The cluster should move from yellow to green shortly afterward.