Building a passwordless OpenSearch logging pipeline with Kubernetes identity

Most Kubernetes log pipelines run on a password: one credential for the log store, copied to every node, shared across clusters and almost never rotated. Kubernetes already gives every pod a signed identity that expires in minutes, and OpenSearch already knows how to check one. We joined the two and ended up with a logging pipeline that has no password anywhere in it, works across a fleet of clusters, and lets each cluster write only its own logs.

The credential nobody wants to rotate

If you have shipped logs from Kubernetes to OpenSearch, you have probably written something like this:

env:
  - name: OPENSEARCH_USER
    value: logshipper
  - name: OPENSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: opensearch-credentials
        key: password

It works on the first try, which is exactly why it sticks around. Follow that password for a while, though. It sits in etcd and in every etcd backup. It is mounted into a DaemonSet, so it is on every node. The second cluster got the same Secret, because a separate user per cluster felt like overhead, and the user can write to every log index, because scoping it felt like overhead too.

Then someone asks when it was last rotated. Rotating it means changing the password in OpenSearch and in every cluster at the same moment, then restarting every shipper, and accepting a gap in the logs if the timing slips. So it does not get rotated, and the longer it lives, the more places it has been copied to.

We wanted the opposite: no credential to store, nothing to rotate, and every cluster limited to its own logs. It turns out both halves of the solution were already deployed. They just had never been introduced.

A pod already has an identity

Every pod in Kubernetes can ask for a token that proves who it is. You request one with a projected volume, naming who the token is for (its audience) and how long it should live:

volumes:
  - name: token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            audience: https://oidc.example.com
            expirationSeconds: 600

Kubelet asks the API server to mint the token, writes it into the pod, and replaces it when it reaches about 80% of its lifetime. The file always holds a valid token, and no human ever touches it.

The token is a JWT signed by the cluster. Decode the middle section and you get claims like these:

{
  "iss": "https://oidc.example.com",
  "sub": "system:serviceaccount:logging:vector",
  "aud": ["https://oidc.example.com"],
  "exp": 1791108000,
  "iat": 1791107400,
  "kubernetes.io": {
    "namespace": "logging",
    "pod": { "name": "vector-7xk2p" },
    "serviceaccount": { "name": "vector" }
  }
}

Three of these matter for logging. iss says which cluster signed it. sub names the workload, in the form system:serviceaccount:<namespace>:<name>. aud says which service the token is meant for, so a token minted for the log store is useless anywhere else.

One detail is worth being precise about. Kubernetes binds these tokens to the pod they were issued for, but only the API server can check that binding. An outside service that verifies the signature sees a valid token until exp. That is why the expiry is short: ten minutes is the minimum Kubernetes allows, and it is the only window in which a leaked token is worth anything.

The cluster publishes its own keys

A signed token is only useful if someone outside the cluster can check the signature. Since version 1.21, Kubernetes does what every OpenID Connect provider does and publishes a discovery document and its public signing keys (ServiceAccount issuer discovery):

kubectl get --raw /.well-known/openid-configuration

{
  "issuer": "https://oidc.example.com",
  "jwks_uri": "https://oidc.example.com/openid/v1/jwks",
  "response_types_supported": ["id_token"],
  "subject_types_supported": ["public"],
  "id_token_signing_alg_values_supported": ["RS256"]
}

That is everything a relying party needs: who the issuer is, and where to get the keys to verify it. As far as the outside world is concerned, a Kubernetes cluster is an OIDC identity provider whose users are workloads.

Teaching OpenSearch to trust a cluster

The other half sits in OpenSearch’s security plugin, which has an openid authenticator. It is usually set up so people can sign in to Dashboards through Okta or Entra ID, but it does not care who the users are. Give it a discovery URL, and it accepts any Bearer token that URL’s keys can verify.

Auth domains live in the plugin’s config.yml, under config.dynamic.authc. We added one next to the existing basic auth domain:

kubernetes:
  order: 1
  http_enabled: true
  transport_enabled: false
  http_authenticator:
    type: openid
    challenge: false
    config:
      openid_connect_url: https://oidc.example.com/.well-known/openid-configuration
      required_audience: https://oidc.example.com
      subject_key: sub
  authentication_backend:
    type: noop

subject_key: sub makes the ServiceAccount the OpenSearch username. required_audience rejects any token from this cluster that was minted for a different service. challenge: false stops this domain from sending a login challenge, which a log shipper never needs. The noop backend means OpenSearch takes the token at its word and looks up no user elsewhere.

Authorization is an ordinary role mapping: a role that can create and write logs-* indices and read cluster health, and nothing more, mapped to the workload’s username:

logs_writer:
  users:
    - "system:serviceaccount:logging:vector"

The fastest way to see it working is to send a token for a ServiceAccount you did not map:

kubectl create token default --audience https://oidc.example.com \
  | sed 's/^/Authorization: Bearer /' \
  | curl -H @- "$OS_URL/_cluster/health"

{"error":{"type":"security_exception","reason":"no permissions for [cluster:monitor/health] and User [name=system:serviceaccount:default:default, backend_roles=[], requestedTenant=null]"},"status":403}

That 403 is good news. OpenSearch fetched the cluster’s keys, verified the signature, checked the audience, and knows exactly who is calling. It just will not let default/default do anything. A 401 instead means the token never got that far, and the usual reason is the next point.

OpenSearch fetches the keys itself, so the issuer’s discovery document and JWKS have to be reachable from the OpenSearch nodes over HTTPS. Most managed Kubernetes services publish their issuer at a public URL, some behind a setting you have to turn on. On a self-managed cluster the API server already serves both paths. Expose them publicly, and allow anonymous reads by binding the built-in system:service-account-issuer-discovery role to unauthenticated users.

Getting the token onto the wire

With both ends ready, we needed a log shipper that could send Authorization: Bearer <token> with every bulk request. We reached for Fluent Bit first, since it is the usual choice, and it could not. As of 5.1.3, its opensearch output authenticates with a username and password or with AWS credentials. It has no setting for a custom header, and an unknown property stops the output from starting. Its generic http output does take headers, but it posts plain records, and OpenSearch’s bulk API needs an action line before every document. We would have been writing a bulk encoder in a Lua filter.

Vector’s elasticsearch sink speaks the bulk API and sends any header you give it, so we switched. It had one catch of its own. Vector reads header values when it loads its configuration, not on every request, and the token on disk changes every eight minutes or so.

Vector can watch its own config file and reload when it changes, so we let a sidecar write the file. The sidecar keeps a template with a placeholder, and whenever the token file’s checksum changes, it renders a fresh config:

render() {
  sed "s|@@TOKEN@@|$(cat "$tok")|" "$tpl" > "$out.tmp" \
    && mv "$out.tmp" "$out"
}

The mv makes the swap atomic, so Vector never reads a half-written file. The rendered config sits on a memory-backed emptyDir, so the token is never written to the node’s disk. Vector runs with --watch-config, and each rotation shows up in its logs as a clean reload:

INFO vector::config::watcher: Configuration file changed.
INFO vector: Vector has reloaded. path=[File("/conf/vector.yaml", None)]

To check that a reload in the middle of a stream does not cost anything, we wrote one log line every five seconds and watched delivery through the rotation of all three shipper pods. It held at six documents every 30 seconds, with no errors and no 401.

A few smaller things caught us out along the way. Vector detects the API version from the cluster root, and OpenSearch 3.x reports a version number that is not an Elasticsearch API version, so we set api_version explicitly. The sink’s startup health check calls cluster health, which is why the writer role is allowed to read it. A shipper that collects its own namespace will happily ship its own error logs, which then fail to send and produce more error logs, so one exclusion pattern leaves the shipper’s own pods out. And Vector’s read positions live on a hostPath rather than inside the pod, so a restart or rollout resumes where the old pod stopped instead of skipping or repeating lines.

One cluster is easy. Every cluster has the same name.

The direct setup worked well for one cluster. Adding a second one broke it, for a reason that is obvious once you see it.

The sub claim is system:serviceaccount:logging:vector in every cluster. Two clusters running the same manifests produce two identities that OpenSearch cannot tell apart. You can add both issuers as separate auth domains, but the role mapping only sees the username and its backend roles, and those are identical. There is no way to say “this ServiceAccount, but only when it comes from prod-eu”, so staging could write into production’s indices.

What we needed was the cluster’s name inside the token, in a claim OpenSearch could map. Dex does exactly that. It supports OAuth 2.0 token exchange: a client posts a token from an upstream provider and gets one of Dex’s own tokens back. Each upstream provider is a connector, and a connector can build a group claim out of the upstream claims as it goes:

connectors:
  - type: oidc
    id: prod-eu
    config:
      issuer: https://oidc.prod-eu.example.com
      clientID: dex
      userNameKey: sub
      claimModifications:
        newGroupFromClaims:
          - claims: [sub]
            delimiter: ":"
            prefix: "prod-eu"

One connector per cluster, each trusting that cluster’s issuer. A connector’s clientID is the audience Dex requires on the incoming token, so the pod projects its ServiceAccount token with audience: dex. The shipper’s sidecar exchanges that token instead of using it directly:

SA_TOKEN=$(kubectl create token vector -n logging --audience dex)

curl https://dex.example.com/token \
  -d client_id=logs-shipper \
  -d grant_type=urn:ietf:params:oauth:grant-type:token-exchange \
  -d connector_id=prod-eu \
  --data-urlencode "subject_token=$SA_TOKEN" \
  -d subject_token_type=urn:ietf:params:oauth:token-type:id_token \
  -d requested_token_type=urn:ietf:params:oauth:token-type:id_token \
  --data-urlencode "scope=openid profile groups"

and gets back a token that carries the cluster’s name:

{
  "iss": "https://dex.example.com",
  "aud": "logs-shipper",
  "name": "system:serviceaccount:logging:vector",
  "groups": ["prod-eu:system:serviceaccount:logging:vector"]
}

Dex sets aud to the client that requested the token, logs-shipper. OpenSearch trusts a single issuer, Dex, with required_audience: logs-shipper, subject_key: name and roles_key: groups. Every group becomes a backend role, and each cluster gets a role that can write only its own index pattern.

We tested the boundary directly. A prod-eu token writing to logs-k8s-prod-eu-2026.10.04 gets a 201 for the document. The same token writing to logs-k8s-prod-us-2026.10.04 gets a 403 and a security_exception for that item. A Dex token issued for any other client is rejected with 401 by required_audience, before any role is checked.

Adding Dex did not add a secret. The Dex client is public, so the exchange needs no client secret, and the only thing the sidecar presents is the pod’s own ten-minute token.

One rule came out of this: map these roles by backend role, never by users. The username is the same in every cluster, so a username mapping on any one role grants that role to all of them.

Keeping one Dex boring

Dex fixed the naming problem, and in doing so became part of the write path of every cluster. If shippers cannot get fresh tokens, logging stops across the whole fleet at once, probably during the incident you most need the logs for. It also decides which cluster every token belongs to: the group that says prod-eu is whatever the Dex config says it is.

That second point decides where Dex runs. It runs in an ops cluster, and its dex namespace there is the one place that controls cluster attribution, so that is the namespace to guard. The ops cluster ships its own logs through the same Dex, like any other. The first point decides how Dex runs:

• One key set, in storage. Dex keeps its signing keys in Kubernetes storage, so restarts, rollouts and every replica share one key set. With in-memory storage, every start would generate new keys, and tokens signed with them are rejected until OpenSearch fetches the new key set.
• Two replicas spread across nodes, with a PodDisruptionBudget that keeps one up during node drains. Rolling updates start a new pod before stopping an old one, and a short preStop sleep keeps a stopping pod serving while the gateway takes it out of rotation. A config change rolls through one pod at a time, and token exchanges keep working throughout.
• Headroom in the shipper. The sidecar exchanges its token every five minutes for one that lasts 30, and keeps the previous token when an exchange fails, so a brief Dex outage does not stop the logs.

The key rotation that takes nine minutes

With Dex running highly available, testing turned up one more failure, and it was the most interesting one. Dex rotates its signing keys on a schedule, and a rotation should be invisible: OpenSearch sees a token signed with a key it does not know, fetches Dex’s key set again, and carries on.

Most of the time that is exactly what happens. But Dex serves its key set with Cache-Control: max-age=600, and OpenSearch respects it. When OpenSearch refetches within ten minutes of its previous fetch, it gets the cached copy back, which does not contain the new key. One rotation is fine, because the cache has usually expired by then. A second key change within ten minutes of the first is not. In testing, a rotation 80 seconds after the previous one was rejected with 401 Authentication finally failed for about nine minutes, and Vector dropped the events it could not send in that time.

We closed it from both ends. Dex rotates weekly (expiry.signingKeys: 168h) rather than every six hours, its default, which makes two rotations in ten minutes very unlikely. And the sidecar does not trust a new token just because Dex issued it. It switches only after OpenSearch has accepted the token, on ten requests in a row:

accepted() {
  for _ in 1 2 3 4 5 6 7 8 9 10; do
    printf 'Authorization: Bearer %s' "$1" \
      | curl -fsS -m 20 -o /dev/null -H @- \
          "$OPENSEARCH_URL/_plugins/_security/authinfo" || return 1
  done
}

Ten, rather than one, because every OpenSearch node keeps its own key cache and refreshes it on its own schedule. A single success can come from the one node that already has the new key. authinfo answers for any signed-in identity, so the write-only role is enough to call it. Until the new token passes, the sidecar keeps sending the previous one, which was signed with the old key and stays valid for 30 minutes, and retries every 30 seconds.

What we ended up with

Here is the pipeline, compared with the password in a Secret we started from:

• No stored credentials. Nothing on any node, in any Secret, or in any etcd backup can write to the log store. Every credential on the write path is minted by the platform, bound to one audience, and expires within 30 minutes.
• Nothing to rotate by hand. Kubelet renews the pod’s token, Dex rotates its own keys, and the sidecar moves to a new token only once OpenSearch accepts it.
• Each cluster writes only its own logs. The cluster name in every token comes from one Dex config, in one namespace you control.
• Real identities. OpenSearch sees a named workload in a named cluster, not a shared logshipper account.

That control has a cost worth stating plainly: whoever can change the dex namespace in the ops cluster can write to every cluster’s indices. Guard it the way you would guard any identity provider.

One credential remains: the OpenSearch admin account used to create roles and mappings. It belongs to whoever runs OpenSearch rather than to the pipeline, and it never has to be copied to a node.

The people reading the logs fit the same model. A SAML or OIDC source on OpenSearch Dashboards maps groups in your identity provider to read-only roles for each cluster, so the team that owns prod-eu sees prod-eu logs and nothing else.

Try it yourself

Everything here runs on open-source components: Kubernetes 1.21 or later, OpenSearch with its security plugin, Vector 0.58 and Dex 2.45. The Kubernetes log shipping guide has both setups end to end, with the complete Vector, Dex and RBAC manifests, plus SAML sign-in for the people reading the logs. The guide’s examples create the OpenSearch side on ClusterNest, but the Kubernetes manifests are the same against any OpenSearch cluster.

If you run OpenSearch yourself, the work that remains is the security plugin: getting the auth domains right in config.yml, in the right order, and applying them with securityadmin.sh after every change.

On ClusterNest Managed OpenSearch, an auth domain is an auth source with a handful of fields in the console, the API or Terraform: the discovery URL, the audience, and the claims that hold the username and groups. Point one at your Dex, or straight at a single cluster’s issuer, alongside the SAML or OIDC login your team already uses. Single sign-on, and every auth source you add, is included on every tier.

Running a setup this doesn’t cover, or a fleet you’re not sure fits? Write to [email protected] and tell us about it.