Building a passwordless OpenSearch logging pipeline with Kubernetes identity

Logging from Kubernetes into OpenSearch often starts with a username and password stored in a Secret. That is straightforward to set up, but the credential spreads across nodes and clusters and almost never gets rotated. Meanwhile, Kubernetes can issue short-lived, signed identities to pods, and OpenSearch can validate them. This article walks through connecting those pieces, then solving the less obvious problem of keeping clusters isolated when their workloads have identical names.

The credential nobody wants to rotate

If you have shipped logs from Kubernetes to OpenSearch, the configuration probably looks familiar:

env:
  - name: OPENSEARCH_USER
    value: logshipper
  - name: OPENSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: opensearch-credentials
        key: password

It works on the first try, which is exactly why it sticks around. The password is stored in etcd and its backups, then mounted into a DaemonSet running across the nodes. When a second cluster comes along, copying the same Secret is the path of least resistance. Before long, one shared account can write to every log index.

Rotation is where the arrangement gets awkward. The password must change in OpenSearch and in every cluster, and the shippers need to pick up the new value at the right time. If the timing slips, logs stop arriving. So the rotation gets put off, and the credential keeps collecting copies.

We wanted to remove the long-lived password entirely and make cluster boundaries enforceable rather than conventional. The pieces were already available in our stack; the work was in getting them to trust each other.

A pod already has an identity

Kubernetes can provide a pod with a token that identifies its ServiceAccount. A projected serviceAccountToken volume lets you specify the intended audience and token lifetime:

volumes:
  - name: token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            audience: https://oidc.example.com
            expirationSeconds: 600

The kubelet asks the API server for the token and projects it into the pod. It replaces the file at roughly 80% of the token’s lifetime, so the file always holds a valid token and no person ever rotates it.

The result is a JWT signed by the cluster. Its payload contains claims along these lines:

{
  "iss": "https://oidc.example.com",
  "sub": "system:serviceaccount:logging:vector",
  "aud": ["https://oidc.example.com"],
  "exp": 1791108000,
  "iat": 1791107400,
  "kubernetes.io": {
    "namespace": "logging",
    "pod": { "name": "vector-7xk2p" },
    "serviceaccount": { "name": "vector" }
  }
}

For this pipeline, three claims do most of the work. iss identifies the issuer, sub identifies the ServiceAccount as system:serviceaccount:<namespace>:<name>, and aud limits which service should accept the token, so a token minted for the log store is useless anywhere else.

Pod binding has a limit worth being precise about. Kubernetes binds a projected token to its pod, but only the API server can check that binding. OpenSearch checks the signature, so it sees a correctly signed token as valid until its exp time. That is why the lifetime is short: ten minutes is the minimum Kubernetes allows, and it is the only window in which a stolen token is worth anything.

The Kubernetes cluster publishes its own keys

For OpenSearch to validate a token, it needs a way to verify the signature without possessing the cluster’s private signing key. Kubernetes 1.21 and later can publish the OIDC discovery document and public signing keys (ServiceAccount issuer discovery):

kubectl get --raw /.well-known/openid-configuration

{
  "issuer": "https://oidc.example.com",
  "jwks_uri": "https://oidc.example.com/openid/v1/jwks",
  "response_types_supported": ["id_token"],
  "subject_types_supported": ["public"],
  "id_token_signing_alg_values_supported": ["RS256"]
}

The discovery document tells OpenSearch who the issuer claims to be and where its public keys live. In this arrangement, Kubernetes acts as the OIDC provider and the workloads running in it are the identities being authenticated.

Teaching OpenSearch to trust a cluster

OpenSearch provides the other half through its Security plugin and its openid authenticator. This authenticator is often used for human sign-in through an identity provider such as Okta or Entra ID, but the same mechanism can validate workload tokens. Configure the discovery URL, and OpenSearch can retrieve the issuer’s keys and verify incoming Bearer tokens.

Authentication domains are configured under config.dynamic.authc in the plugin’s config.yml. We added an OIDC domain alongside the existing basic-auth domain:

kubernetes:
  order: 1
  http_enabled: true
  transport_enabled: false
  http_authenticator:
    type: openid
    challenge: false
    config:
      openid_connect_url: https://oidc.example.com/.well-known/openid-configuration
      required_audience: https://oidc.example.com
      subject_key: sub
  authentication_backend:
    type: noop

Here, subject_key: sub tells OpenSearch to use the ServiceAccount subject as the username. required_audience rejects tokens minted for a different audience, while challenge: false prevents an interactive authentication challenge that a shipper cannot answer. The noop backend means the verified identity is used directly rather than looked up in another user store.

Authentication establishes who the caller is; role mapping decides what it can do. We gave the shipper only the permissions it needs: create and write logs-* indices and read cluster health. The mapping associates that role with the workload’s username:

logs_writer:
  users:
    - "system:serviceaccount:logging:vector"

A useful smoke test is to send a valid token for a ServiceAccount that has no role mapping:

kubectl create token default --audience https://oidc.example.com \
  | sed 's/^/Authorization: Bearer /' \
  | curl -H @- "$OS_URL/_cluster/health"

{"error":{"type":"security_exception","reason":"no permissions for [cluster:monitor/health] and User [name=system:serviceaccount:default:default, backend_roles=[], requestedTenant=null]"},"status":403}

The 403 is the expected result: OpenSearch has authenticated system:serviceaccount:default:default, but that identity has no permission to read cluster health. A 401 means the token never got that far. The usual cause is an issuer URL or audience that does not match.

OpenSearch retrieves the discovery document and JWKS itself, which means both endpoints must be reachable from every OpenSearch node over HTTPS. Many managed Kubernetes services expose the issuer publicly, though some require it to be enabled. On a self-managed cluster, the API server serves these endpoints; make them reachable and permit anonymous reads by binding the built-in system:service-account-issuer-discovery role to unauthenticated users.

Getting the token onto the wire

Once OpenSearch could validate the token, the remaining question was how to attach it to each bulk request. We reached for Fluent Bit first, and it could not send the header. As of 5.1.3, its OpenSearch output supports username/password and AWS authentication, with no setting for a custom header. The generic HTTP output accepts custom headers, but it does not format records for the OpenSearch bulk API, which expects an action line before each document. Making that work would have meant implementing the bulk encoding ourselves in a Lua filter.

Vector fit better: its elasticsearch sink understands the bulk API and lets us set custom headers. There was one wrinkle. Vector reads those header values when it loads its configuration; it does not reread the token file for every request. Since Kubernetes rotates the token, a static configuration would eventually start sending an expired value.

Vector can watch its configuration file and reload it when it changes, so we put a small sidecar in charge of rendering that file. It starts from a template and replaces a placeholder whenever the projected token changes:

render() {
  sed "s|@@TOKEN@@|$(cat "$tok")|" "$tpl" > "$out.tmp" \
    && mv "$out.tmp" "$out"
}

The final mv is atomic, so Vector never reads a partially written file: it sees either the old configuration or the new one. We keep the rendered configuration on a memory-backed emptyDir to avoid writing the token to node storage. Vector runs with --watch-config and logs each successful reload:

INFO vector::config::watcher: Configuration file changed.
INFO vector: Vector has reloaded. path=[File("/conf/vector.yaml", None)]

We wanted to know whether a reload would interrupt delivery, so we emitted one test line every five seconds while all three shipper pods rotated their tokens. Delivery stayed at six documents per 30-second interval, with no send errors and no 401 responses.

We also had to account for a few practical details. Vector tries to infer the API version from the cluster root, but OpenSearch 3.x reports a version that does not map cleanly to an Elasticsearch API version, so we set api_version explicitly. The sink’s startup health check calls the cluster-health endpoint, which is why the writer role needs that read permission. We exclude the shipper’s own pods from collection; otherwise, its failed-send messages can be collected and sent back into the same failing pipeline. Finally, Vector’s read positions are stored on a hostPath, so a restarted or replaced pod can resume from its previous position instead of skipping or duplicating lines.

One Kubernetes cluster is easy. Multiple are a problem

The direct setup worked for a single cluster. The problem only appeared when we added another cluster.

The subject claim is identical in both places: system:serviceaccount:logging:vector. If the clusters use the same manifests, OpenSearch sees the same username from each one. Separate auth domains can trust separate issuers, but the role mapping still works with the username and backend roles, which do not distinguish these clusters. Without another claim to map, staging could write into production’s indices.

We needed the identity to carry the cluster name as well as the ServiceAccount. Dex gives us a way to add that context. With OAuth 2.0 token exchange, a client presents a token from an upstream identity provider and receives a Dex-issued token. Each cluster gets its own connector, and the connector can derive a group claim from the incoming claims:

connectors:
  - type: oidc
    id: prod-eu
    config:
      issuer: https://oidc.prod-eu.example.com
      clientID: dex
      userNameKey: sub
      claimModifications:
        newGroupFromClaims:
          - claims: [sub]
            delimiter: ":"
            prefix: "prod-eu"

We configure one connector for each cluster and point it at that cluster’s issuer. The connector’s clientID is the audience expected on the incoming token, so the pod requests its ServiceAccount token for dex. The sidecar then exchanges that token with Dex rather than sending it straight to OpenSearch:

SA_TOKEN=$(kubectl create token vector -n logging --audience dex)
curl https://dex.example.com/token \
  -d client_id=logs-shipper \
  -d grant_type=urn:ietf:params:oauth:grant-type:token-exchange \
  -d connector_id=prod-eu \
  --data-urlencode "subject_token=$SA_TOKEN" \
  -d subject_token_type=urn:ietf:params:oauth:token-type:id_token \
  -d requested_token_type=urn:ietf:params:oauth:token-type:id_token \
  --data-urlencode "scope=openid profile groups"

Dex returns a token with a group that identifies both the cluster and the ServiceAccount:

{
  "iss": "https://dex.example.com",
  "aud": "logs-shipper",
  "name": "system:serviceaccount:logging:vector",
  "groups": ["prod-eu:system:serviceaccount:logging:vector"]
}

The returned token’s audience is logs-shipper, the client that requested it. OpenSearch now needs to trust only Dex, with required_audience: logs-shipper, subject_key: name, and roles_key: groups. OpenSearch maps the group claim to backend roles, allowing us to give each cluster write access only to its own index pattern.

We verified the boundary with actual writes. A prod-eu token writing to logs-k8s-prod-eu-2026.10.04 gets a 201. Send that same token to logs-k8s-prod-us-2026.10.04, and OpenSearch returns 403 with a security_exception. A Dex token minted for a different client fails the required_audience check with 401, before role mapping comes into play.

Dex does not introduce another long-lived secret here. The client is public, so the exchange uses no client secret; the sidecar presents only the pod’s short-lived ServiceAccount token.

There is one easy-to-miss rule: map these permissions through backend roles, not through users. The username remains the same across clusters, so a direct username mapping would grant that role to every cluster using that ServiceAccount name.

Making Dex reliable

Dex solves the identity collision, but it also becomes a dependency for every cluster’s log pipeline. If shippers cannot obtain fresh tokens, the whole fleet stops delivering logs at once. That is most likely during the incident you need the logs for. Dex also controls cluster attribution: a token’s prod-eu group is determined by the connector configuration.

For that reason, Dex runs in an ops cluster. Its dex namespace is a control point for cluster identity and needs to be protected accordingly. The ops cluster’s own shippers use the same path as everyone else. Availability matters just as much, so we set Dex up to tolerate routine pod and node changes:

• One key set, in storage. Dex keeps its signing keys in Kubernetes storage, so restarts, rollouts and every replica share one key set. With in-memory storage, every start would generate new keys, and tokens signed with them are rejected until OpenSearch fetches the new key set.
• Two replicas spread across nodes, with a PodDisruptionBudget that keeps one up during node drains. Rolling updates start a new pod before stopping an old one, and a short preStop sleep keeps a stopping pod serving while the gateway takes it out of rotation. A config change rolls through one pod at a time, and token exchanges keep working throughout.
• Headroom in the shipper. The sidecar exchanges its token every five minutes for one that lasts 30, and keeps the previous token when an exchange fails, so a brief Dex outage does not stop the logs.

The key rotation that takes nine minutes

Even with Dex running across replicas, we found a failure case during testing. Key rotation should be transparent: when OpenSearch encounters a token signed by a new key, it fetches Dex’s JWKS and retries validation with the updated key set.

Usually, that is what happens. The complication is caching: Dex serves JWKS with Cache-Control: max-age=600, and OpenSearch honors that ten-minute lifetime. If OpenSearch refetches within ten minutes of its previous fetch, it gets its own cached copy back, which does not contain the new key. One rotation is fine, because the cache has usually expired by then. A second rotation within ten minutes of the first is not. In our test, rotating again 80 seconds later caused 401 Authentication finally failed responses for roughly nine minutes, and Vector dropped events it could not deliver.

We addressed this on both sides. Dex now rotates its signing keys weekly (expiry.signingKeys: 168h) instead of every six hours, its default, making overlapping rotations much less likely. The sidecar also verifies that OpenSearch accepts a newly issued token before replacing the current one. It requires ten successful checks in a row:

accepted() {
  for _ in 1 2 3 4 5 6 7 8 9 10; do
    printf 'Authorization: Bearer %s' "$1" \
      | curl -fsS -m 20 -o /dev/null -H @- \
          "$OPENSEARCH_URL/_plugins/_security/authinfo" || return 1
  done
}

We use ten checks because each OpenSearch node has its own key cache and refreshes on its own schedule. One successful request might simply have reached a node that already knows the new key. The authinfo endpoint reports the authenticated identity, so the shipper’s limited role can call it. Until all ten checks pass, the sidecar keeps sending the previous token, which stays valid for 30 minutes, and retries the new token every 30 seconds.

What we ended up with

Compared with the password in a Secret we started from:

• No stored credentials. Nothing on any node, in any Secret, or in any etcd backup can write to the log store. Every credential on the write path is minted by the platform, bound to one audience, and expires within 30 minutes.
• Nothing to rotate by hand. Kubelet renews the pod’s token, Dex rotates its own keys, and the sidecar moves to a new token only once OpenSearch accepts it.
• Each cluster writes only its own logs. The cluster name in every token comes from one Dex config, in one namespace you control.
• Real identities. OpenSearch sees a named workload in a named cluster, not a shared logshipper account.

That control has a cost worth stating plainly: whoever can change the dex namespace in the ops cluster can write to every cluster’s indices. Guard it the way you would guard any identity provider.

The OpenSearch administrator credential still exists for managing roles and mappings. It belongs to the operators of OpenSearch, not to the log pipeline, and it does not need to be distributed to worker nodes.

The same separation can apply to people using OpenSearch Dashboards. Configure SAML or OIDC and map identity-provider groups to read-only roles scoped to each cluster’s indices. The team responsible for prod-eu can then read those logs without gaining access to other clusters.

Try it yourself

The setup uses open-source components: Kubernetes 1.21 or later, OpenSearch with the Security plugin, Vector 0.58, and Dex 2.45. The Kubernetes log shipping guide walks through both configurations and includes the Vector, Dex, and RBAC manifests. Its OpenSearch examples use ClusterNest, but the Kubernetes-side manifests work with other OpenSearch deployments too.

If you operate OpenSearch yourself, most of the remaining work is in the Security plugin configuration: put the auth domains in the right order in config.yml and apply changes with securityadmin.sh.

With ClusterNest Managed OpenSearch, you can configure an auth domain as an auth source in the console, API, or Terraform. Set its discovery URL, audience, and the claims used for the username and groups. You can point it at Dex for a multi-cluster fleet or directly at one cluster’s issuer, alongside your existing SAML or OIDC login. Single sign-on, and every auth source you add, is included on every tier.

If your setup differs from this one, or you want to check how it would work across your fleet, get in touch at [email protected].