Pushing Metrics into ClusterNest Managed Prometheus

Every cluster comes with a write credential, scoped to exactly the push paths and nothing else

• it can’t run queries, read rules, or reach the alertmanager config API. Use it for any remote_write-speaking agent (Prometheus itself in agent mode, Grafana Agent, an OpenTelemetry Collector’s Prometheus remote-write exporter).

Fetch the write password with GET /cluster/prometheus/$CLUSTER_ID/credentials, then point remote_write at the push endpoint - note this is the cluster’s root URL, not /prometheus-prefixed (that prefix is for the query API only):

remote_write:
  - url: https://$PROMETHEUS_HOST/api/v1/push
    basic_auth:
      username: write
      password: $WRITE_PASSWORD

Remote write 2.0

The push endpoint accepts remote write 2.0 as well as 1.0. 2.0 sends each series’ labels and metadata once per request instead of repeating them, which cuts egress sharply for large label sets - about two thirds on two of our own clusters running the full kube-prometheus stack. Upstream still marks 2.0 experimental, so try it on a non-production Prometheus first. It needs Prometheus 2.54 or later; set the message on each remote_write entry:

remote_write:
  - url: https://$PROMETHEUS_HOST/api/v1/push
    protobuf_message: io.prometheus.write.v2.Request
    basic_auth:
      username: write
      password: $WRITE_PASSWORD

With the Prometheus Operator (kube-prometheus), set messageVersion: V2.0 on the entry in spec.remoteWrite instead. Leave the field out to keep sending 1.0.

Legacy path

A legacy path also works, for older clients that expect it:

https://$PROMETHEUS_HOST/api/prom/push

Verifying it’s working

Push a sample series with curl (a zero-length snappy-compressed WriteRequest - snappy’s block format is a length prefix followed by that many compressed bytes, so a length of zero is just the single byte \x00 - the minimal valid payload, just to confirm auth and connectivity without needing a client library). A NUL byte can’t be passed as a shell argument directly (it truncates to nothing), so write it to a file first:

printf '\x00' > empty.snappy

curl -i -X POST "https://$PROMETHEUS_HOST/api/v1/push" \
  -u "write:$WRITE_PASSWORD" \
  -H "Content-Type: application/x-protobuf" \
  -H "Content-Encoding: snappy" \
  -H "X-Prometheus-Remote-Write-Version: 0.1.0" \
  --data-binary @empty.snappy

A 200 confirms the credential and endpoint are correct. For a real payload, point an actual Prometheus/Grafana Agent at the endpoint above and check its own remote_write send metrics (prometheus_remote_storage_succeeded_samples_total) rather than hand-building protobuf.

Once data has been pushed, confirm it’s queryable with the readonly or admin credential (see Querying + Grafana):

curl -u "readonly:$READONLY_PASSWORD" \
  "https://$PROMETHEUS_HOST/prometheus/api/v1/query?query=up"

Retention

How long pushed data is kept is set per cluster (retention_period - 90d by default, settable at create time or changed later, same as any other field - see the console/Terraform/ raw API tabs on Getting started for the three ways to reach a cluster’s settings). Data older than the retention window is deleted, not archived - lower it only if that’s the intended behavior.