Configure Logging with ClusterNest Managed OpenSearch

Set up centralized logging with Fluent Bit and ClusterNest Managed OpenSearch: provision a cluster via the Console or Terraform, forward logs with Fluent Bit, and visualize them in OpenSearch Dashboards.

Console

Log in to ClusterNest Console. Navigate to OpenSearch from the Services section and launch a cluster.

• Enter a unique name for your OpenSearch cluster.
• Select a performance tier based on resource needs: Basic, Standard, or Advanced.
• Choose the required OpenSearch version (optional).
• Click Create Cluster to deploy.

Save the credentials securely for configuration and accessing OpenSearch Dashboards later.

Index Template

After the cluster is available, open OpenSearch Dashboards from the cluster details page. Log in using the saved credentials.

From the left sidebar, navigate to Stack Management > Index Management > Templates. Create a new index template named fluent-bit with the following settings:

• Index patterns: fluent-bit-*
• Priority: 1
• Number of shards: 1
• Number of replicas: 1
• Click Create template to save.

Terraform

We’ll begin by provisioning an OpenSearch cluster. This cluster will serve as the central destination for all logs collected by Fluent Bit.

Providers

terraform {
  required_providers {
    clusternest = {
      source  = "tf.clusternest.com/clusternest/clusternest"
      version = ">=1.1.0"
    }
    opensearch = {
      source  = "opensearch-project/opensearch"
      version = ">= 2.2.0"
    }
  }
}

provider "clusternest" {
  email        = <EMAIL>
  app_password = <APP_PASSWORD>
}

Outputs

Use outputs to get the username, password and cluster API endpoint. Save them securely for configuration and accessing the Dashboard later.

data "clusternest_opensearch_credentials" "this" {
  cluster_id = clusternest_opensearch.this.id
}

output "endpoint" {
  value = clusternest_opensearch.this.url
}

output "username" {
  value = data.clusternest_opensearch_credentials.this.username
}

output "password" {
  value     = data.clusternest_opensearch_credentials.this.password
  sensitive = true
}

Cluster

Create the cluster and index template.

resource "clusternest_opensearch" "this" {
  name                            = "cluster1"
  tier                            = "basic"
  organization_id                 = <ORG_ID>
}

provider "opensearch" {
  url      = clusternest_opensearch.this.url
  username = data.clusternest_opensearch_credentials.this.username
  password = data.clusternest_opensearch_credentials.this.password
}

resource "opensearch_index_template" "fluent-bit" {
  name = "fluent-bit"
  body = <<EOF
{
  "index_patterns": ["fluent-bit-*"],
  "template": {
    "settings": {
      "index": {
        "number_of_shards": "1",
        "number_of_replicas": "1"
      }
    }
  },
  "_meta": {
    "flow": "simple"
  },
  "priority": 1
}
EOF
}

Note

If you encounter authentication errors, check if the app password is valid.

Fluent Bit

Shipping from Kubernetes without a password

The Kubernetes tabs below keep the cluster password in a Secret. Ship Kubernetes logs without passwords builds the same pipeline with no static credential: each log shipper signs in with the identity Kubernetes already gives its pod.

Linux

Create/Edit fluent-bit.conf

[SERVICE]
    Flush        5
    Log_Level    info
    Parsers_File parsers.conf
[INPUT]
    Name             tail
    Path             <logs_path>/*.log
    Tag              *
    Refresh_Interval 2
[OUTPUT]
    Name             opensearch
    Match            *
    Port             443
    tls              On
    Host             <CLUSTER_HOST>
    HTTP_User        <CLUSTER_USERNAME>
    HTTP_Passwd      <CLUSTER_PASSWORD>
    Logstash_Format  On
    Logstash_Prefix  fluent-bit
    Replace_Dots     On
    Retry_Limit      False
    Suppress_Type_Name On
    Buffer_Size      False
    Generate_ID      On

Fill in <CLUSTER_HOST>, <CLUSTER_USERNAME>, and <CLUSTER_PASSWORD> from the Terraform outputs above.

• Refresh_Interval: looks for new log files every 2 seconds instead of the default 60, so a file that rotates quickly is read before it’s rotated away.
• Buffer_Size: reads OpenSearch’s full bulk response. With the default 512 KB limit, a large batch’s response is cut off and Fluent Bit resends logs that were already stored.
• Generate_ID: gives each record an ID derived from its content, so a record sent again on a retry isn’t stored twice.

Create/Edit parsers.conf

[PARSER]
    Name        cri
    Format      regex
    Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Save the config and run:

• sudo systemctl enable --now fluent-bit
• sudo systemctl status fluent-bit

Run the following to generate a sample log and verify ingestion:

echo "Test log entry" | sudo tee -a <logs_path>/test.log
sudo journalctl -u fluent-bit -n 5

Verify

sudo systemctl is-enabled fluent-bit

This should return: enabled

Docker Compose

To collect logs from all containers on your system, Docker can use the Fluentd logging driver, which captures the container logs and sends it over the network to Fluent Bit container.

Create fluent-bit.conf

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
[INPUT]
    Name         forward
    Listen       0.0.0.0
    Port         24224
[OUTPUT]
    Name               opensearch
    Match              *
    Port               443
    tls                On
    Host               <CLUSTER_HOST>
    HTTP_User          <CLUSTER_USERNAME>
    HTTP_Passwd        <CLUSTER_PASSWORD>
    Logstash_Format    On
    Logstash_Prefix    fluent-bit
    Suppress_Type_Name On
    Retry_Limit        False
    Buffer_Size        False
    Generate_ID        On

• Buffer_Size: reads OpenSearch’s full bulk response. With the default 512 KB limit, a large batch’s response is cut off and Fluent Bit resends logs that were already stored.
• Generate_ID: gives each record an ID derived from its content, so a record sent again on a retry isn’t stored twice.

Create docker-compose.yaml

services:
  fluent-bit:
    image: fluent/fluent-bit:3.2
    container_name: fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
    ports:
      - "24224:24224"
    networks:
      - logging
networks:
  logging:

The networks section puts Fluent Bit on a Docker network called logging, so other containers on that network can send it their logs.

Add a logging section in your app’s Docker configuration to send container logs to Fluent Bit. This uses the Fluentd logging driver to forward logs at the specified fluentd-address.

Example app:

myapp:
  image: python:3.11
  container_name: myapp
  working_dir: /app
  volumes:
    - ./app.py:/app/app.py
  command: ["python", "app.py"]
  logging:
    driver: "fluentd"
    options:
      fluentd-address: "localhost:24224"
      tag: "docker.{{.Name}}"
  depends_on:
    - fluent-bit
  networks:
    - logging

Each log is tagged with the container name, allowing easy identification and organization. Add a container_name mapping to your OpenSearch index template so you can query logs by container, and repeat the logging block above for every container you want to monitor.

Build and run the setup with docker compose up -d

Kubernetes (Manifests)

To collect logs from all relevant nodes or applications in your Kubernetes cluster, Fluent Bit is typically deployed as a DaemonSet.

In our setup, we’ll create a Fluent Bit DaemonSet that mounts the host’s log directories (/var/log) so Fluent Bit can access container logs.

1. DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:3.0
          imagePullPolicy: IfNotPresent
          args: ["-c", "/fluent-bit/etc/fluent-bit.conf", "-v"]
          env:
            - name: OPENSEARCH_USER
              valueFrom:
                secretKeyRef:
                  name: os-secrets
                  key: OPENSEARCH_USER
            - name: OPENSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: os-secrets
                  key: OPENSEARCH_PASSWORD
            - name: OPENSEARCH_HOST
              valueFrom:
                secretKeyRef:
                  name: os-secrets
                  key: OPENSEARCH_HOST
          resources:
            limits:
              memory: 200Mi
              cpu: 200m
            requests:
              memory: 100Mi
              cpu: 100m
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: config
              mountPath: /fluent-bit/etc/
      serviceAccountName: fluent-bit
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: config
          configMap:
            name: fluent-bit-config

2. ConfigMap

Create config-map.yaml. This configuration collects logs from all application pods across the cluster while excluding system and logging namespaces. It uses the CRI parser for Kubernetes container logs and forwards them to OpenSearch.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
    [INPUT]
        Name              tail
        Tag               kube.*
        Skip_Long_Lines   On
        Parser            cri
        Path              /var/log/containers/*.log
        Exclude_Path      /var/log/containers/*_kube-system_*.log,/var/log/containers/*_logging_*.log
        Refresh_Interval  2
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Kube_Tag_Prefix     kube.var.log.containers.
    [OUTPUT]
        Name            opensearch
        Match           *
        Host            ${OPENSEARCH_HOST}
        Port            443
        tls             On
        HTTP_User       ${OPENSEARCH_USER}
        HTTP_Passwd     ${OPENSEARCH_PASSWORD}
        Logstash_Format On
        Logstash_Prefix fluent-bit
        Suppress_Type_Name On
        Replace_Dots    On
        Retry_Limit     False
        Buffer_Size     False
        Generate_ID     On

  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

• Path: captures logs from all containers (*.log).
• Exclude_Path: avoids logging system and logging namespace pods to reduce noise.
• Refresh_Interval: looks for new log files every 2 seconds instead of the default 60. Kubelet rotates a container’s log at 10 MiB by default, so a busy pod can fill and rotate a file between two 60-second checks, and Fluent Bit never reads it.
• [FILTER]: fetches metadata from the Kubernetes API and adds namespace, pod, and container metadata to each log entry. Merge_Log turns a JSON log line into top-level fields, and Keep_Log Off drops the original string once it’s parsed, so log only ever holds plain-text lines.
• Buffer_Size: reads OpenSearch’s full bulk response. With the default 512 KB limit, a large batch’s response is cut off and Fluent Bit resends logs that were already stored, which happens when it catches up on a backlog.
• Generate_ID: gives each record an ID derived from its content. When OpenSearch rejects part of a batch, Fluent Bit retries the whole batch, and without this the records that were accepted the first time are stored twice.

3. Secrets & Environments

Create a Kubernetes Secret for Fluent Bit credentials: secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: os-secrets
  namespace: logging
type: Opaque
stringData:
  OPENSEARCH_USER: "<your-username>"
  OPENSEARCH_PASSWORD: "<your-password>"
  OPENSEARCH_HOST: "<cluster-host>"

4. Namespace

Create namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: logging

5. Service Account

Create a dedicated ServiceAccount for Fluent Bit in the logging namespace

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging

6. ClusterRole

Define the permissions Fluent Bit requires

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - pods
  verbs: ["get", "list", "watch"]

7. ClusterRoleBinding

Bind the ClusterRole to the ServiceAccount so it can act with the defined permissions

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
  namespace: logging
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: logging
roleRef:
  kind: ClusterRole
  name: fluent-bit
  apiGroup: rbac.authorization.k8s.io

This binding gives Fluent Bit cluster-wide access to list and watch pods and namespaces.

8. Index template

Update the OpenSearch index template and add a mappings field

"mappings": {
  "properties": {
    "kubernetes": {
      "properties": {
        "namespace_name": { "type": "keyword" },
        "pod_name":       { "type": "keyword" },
        "container_name": { "type": "keyword" },
        "labels": {
          "properties": {
            "app": { "type": "keyword" }
          }
        }
      }
    }
  }
}

Once the manifests are ready, deploy everything using kubectl apply -f . After deploying Fluent Bit, create an OpenSearch index pattern fluent-bit-* in the dashboard to visualize all collected logs. You can filter logs based on Kubernetes attributes such as namespace, pod name, or app name.

Kubernetes (Helm)

Fluent Bit publishes an official Helm chart that creates the ServiceAccount, RBAC, and DaemonSet for you.

Add the chart repo:

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

Create the namespace and a Secret with your OpenSearch credentials:

kubectl create namespace logging
kubectl create secret generic os-secrets -n logging \
  --from-literal=OPENSEARCH_HOST=<cluster-host> \
  --from-literal=OPENSEARCH_USER=<your-username> \
  --from-literal=OPENSEARCH_PASSWORD=<your-password>

Create values.yaml, overriding the chart’s default input and output:

config:
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
        Refresh_Interval 2

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  outputs: |
    [OUTPUT]
        Name            opensearch
        Match           kube.*
        Host            ${OPENSEARCH_HOST}
        Port            443
        tls             On
        HTTP_User       ${OPENSEARCH_USER}
        HTTP_Passwd     ${OPENSEARCH_PASSWORD}
        Logstash_Format On
        Logstash_Prefix fluent-bit
        Suppress_Type_Name On
        Replace_Dots    On
        Retry_Limit     False
        Buffer_Size     False
        Generate_ID     On

env:
  - name: OPENSEARCH_HOST
    valueFrom:
      secretKeyRef:
        name: os-secrets
        key: OPENSEARCH_HOST
  - name: OPENSEARCH_USER
    valueFrom:
      secretKeyRef:
        name: os-secrets
        key: OPENSEARCH_USER
  - name: OPENSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: os-secrets
        key: OPENSEARCH_PASSWORD

The inputs block is the chart’s default with Refresh_Interval 2 added, and the output sets Buffer_Size and Generate_ID. The Kubernetes (Manifests) tab explains all three.

Install the chart:

helm install fluent-bit fluent/fluent-bit -n logging -f values.yaml

The chart’s default kubernetes filter already adds namespace, pod, and container metadata to each log, same as the Kubernetes (Manifests) tab. Add the index template mapping from that tab’s Index template step, then create an OpenSearch index pattern fluent-bit-* in the dashboard to visualize the logs.

Dashboard

Open OpenSearch Dashboards. From the left sidebar, navigate to Discover. Create an index pattern using fluent-bit-*. You should start seeing logs in Discover.

Once verified, you have a complete centralized logging setup powered by Fluent Bit and ClusterNest Managed OpenSearch.

Debugging

For debugging Fluent Bit, update config-map.yaml

Add

[OUTPUT]
    Name  stdout
    Match *

Update

[SERVICE]
    Log_Level debug

For Linux, run

sudo journalctl -u fluent-bit

Args in DaemonSet

args: ["-c", "/fluent-bit/etc/fluent-bit.conf", "-v", "--trace-input","--trace-output"]

Retention Policy

Fluent Bit creates a new fluent-bit-* index every day, so without cleanup they grow indefinitely. Set up an Index State Management (ISM) policy to delete old indices automatically.

Console

Open Dev Tools from the left sidebar and run:

PUT _plugins/_ism/policies/fluent-bit-retention
{
  "policy": {
    "description": "Delete fluent-bit logs after 7 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "7d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [{ "delete": {} }],
        "transitions": []
      }
    ],
    "ism_template": [
      { "index_patterns": ["fluent-bit-*"], "priority": 100 }
    ]
  }
}

Terraform

resource "opensearch_ism_policy" "fluent-bit" {
  policy_id = "fluent-bit-retention"
  body      = <<EOF
{
  "policy": {
    "description": "Delete fluent-bit logs after 7 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "7d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [{ "delete": {} }],
        "transitions": []
      }
    ],
    "ism_template": [
      { "index_patterns": ["fluent-bit-*"], "priority": 100 }
    ]
  }
}
EOF
}

The ism_template field attaches the policy to every index matching fluent-bit-*, including ones created after the policy exists. Adjust min_index_age to your preferred retention.