What 100 MB of logs costs in OpenSearch
We shipped exactly 100 MB of logs into OpenSearch, once with Fluent Bit and once with Vector, and measured every byte that landed on disk. One copy came to about 1.5 times the raw size, the replica doubled that, and the search index, not the logs, turned out to be the biggest part of the bill.
The number every log budget starts with
Somebody always asks it before a logging cluster gets sized: we produce this many gigabytes of logs a day, so how much disk do we need?
The usual answers are folklore. “Logs compress, so less than the raw size.” “Indexing adds overhead, so budget double.” Both are true of some setup somewhere. Neither tells you what your shipper, your metadata and your mappings do to your bytes.
So we measured it, on the most common pipeline there is: container logs read off a Kubernetes node, enriched with pod metadata, and pushed into OpenSearch with default settings.
The test bench
A generator pod wrote exactly 100,000,209 bytes to stdout at about 1 MB a second: 542,202 lines from a fixed random seed, so every run saw the same lines. The mix aims at a typical service:
- 40 percent nginx-style access lines
- 40 percent JSON application logs with trace IDs, user IDs and durations
- 20 percent plain text application lines
On the other side, a three-data-node OpenSearch 3.4 cluster with no index templates, so every index got the defaults: one primary shard, one replica, dynamic mapping, the default codec.
Two shippers, each in its usual Kubernetes configuration:
- Fluent Bit 4.2.3: the
tailinput with thecriparser, then thekubernetesfilter withMerge_Log On, which lifts JSON log lines into top-level fields. - Vector 0.53.0: the
kubernetes_logssource into theelasticsearchsink, with no transforms.
Each ran alone against its own index, so neither competed with the other for node or cluster resources. After every run we checked that the document count matched the line count exactly, force-merged the index to one segment so leftover merge files would not count, and read the size from the index stats. Then we ran du on every data node’s index directory, and it agreed with the stats to within 1.5 KB per shard.
About 1.5 times, then double
| Shipper | One copy | With the replica | One copy vs raw |
|---|---|---|---|
| Fluent Bit | 151.8 MB | 304.2 MB | 1.52× |
| Vector | 156.7 MB | 313.3 MB | 1.57× |
Logs compress well, and the index still came out bigger than the input. With the replica that every production cluster should run, 100 MB of logs took about 3 times its size on disk.
Here is one of Fluent Bit’s documents, trimmed:
{
"@timestamp": "2026-02-24T12:16:36.573Z",
"level": "INFO",
"msg": "payment authorized",
"service": "billing",
"trace_id": "62615b67acb100eb7837ea228b0df2e1",
"user_id": 94952,
"kubernetes": {
"pod_name": "loggen",
"namespace_name": "log-storage-test",
"pod_id": "aecd3fd9-f8fa-41ff-a365-7fcacbeff192",
"pod_ip": "198.51.100.24",
"host": "worker-1",
"container_image": "docker.io/library/python:3.12-alpine",
"container_hash": "docker.io/library/python@sha256:1b66...",
"docker_id": "994fa6ad13f0f1cfaa413a2ae2eb3363...",
"labels": { "app": "loggen" }
}
}The log line is the top half. The bottom half rides along on every line the pod writes. Vector’s documents carry more of it: the pod’s labels and IP, but also every label on the node and the namespace, the container ID and the path of the log file.
What the metadata costs
One gotcha turned up in the first sample document. Fluent Bit copies pod annotations into every record, and our generator pod had been created with kubectl apply, which stores the whole pod spec in the kubectl.kubernetes.io/last-applied-configuration annotation. That added about 600 bytes of JSON to each log line. Pods created by a Deployment don’t carry that annotation, but anything that is applied directly does, and so does any pod with large annotations of its own (sidecar injectors and scrape configs both add them). Setting Annotations Off on Fluent Bit’s kubernetes filter keeps them out of the record. Every number in this post comes from a generator pod created without it.
To measure the metadata itself, we ran both shippers again without it. Without metadata, a shipper doesn’t need Kubernetes at all, so the generator wrote the same seeded lines to a file on the node, and each shipper read that file from the start. Fluent Bit used tail with a json parser filter in place of Merge_Log. Vector used its file source.
| Shipper | With metadata | Without | Metadata share |
|---|---|---|---|
| Fluent Bit | 151.8 MB | 129.5 MB | 15% |
| Vector | 156.7 MB | 126.2 MB | 19% |
The runs without metadata also lose the small fields the container runtime’s log format adds (stream, time), so the share is a close estimate rather than an exact one. Vector’s share is the larger one because its metadata is: the node and namespace labels repeat on every line. A remap with del(.kubernetes.node_labels) removes the node’s labels if you never filter on them.
Where the bytes go
Even with no metadata, 100 MB of logs became 126 to 130 MB. OpenSearch can tell you why: the segment stats split an index by Lucene file type.
GET /vector-nometa/_stats/segments?include_segment_file_sizes=true| Part | Vector | Fluent Bit |
|---|---|---|
Stored documents (fdt) |
50.8 MB | 43.5 MB |
Term dictionary (tim) |
36.3 MB | 39.6 MB |
Doc values (dvd) |
19.9 MB | 28.8 MB |
Postings and positions (doc, pos) |
15.2 MB | 10.6 MB |
| Points, norms and the rest | 4.0 MB | 7.0 MB |
The stored documents are the logs themselves. Compression takes the 100 MB of text down to 44 to 51 MB. Everything else is the machinery that makes logs searchable, and it is bigger than the logs.
Three things drive it.
Every string is indexed twice. With no template, dynamic mapping turns each string field into this:
{
"type": "text",
"fields": {
"keyword": { "type": "keyword", "ignore_above": 256 }
}
}The text field is split into words for full-text search. The keyword subfield stores the whole value again as one term, plus a doc-values column for sorting and aggregations. Vector keeps each log line as a single message string, so for the 378,322 lines under 256 characters, the entire line went into the index a second and a third time.
Unique values don’t compress. Trace IDs, span IDs, IP addresses and user IDs are close to unique per line. The term dictionary shares prefixes between similar terms, and random hex has none to share.
Parsing adds fields. In the runs without metadata, Fluent Bit’s JSON parsing produced 12 mapped fields, 7 of them strings with a keyword copy. Vector’s index had 5, with 4 strings. Each string field gets its own text index, keyword and doc values, and each number gets a points index. That is why Fluent Bit’s index is the larger one here even though it stores less: you pay for being able to search by service or sort by duration_ms.
With metadata on, the two effects meet in the middle. Fluent Bit’s parsed fields cost more to index, Vector’s heavier metadata costs more to store, and the two indexes end up within 5 MB of each other.
Making it smaller
The cheapest change is the codec. We reindexed every index into a copy with index.codec: best_compression and merged it again.
| Index | Default codec | best_compression |
|---|---|---|
| Fluent Bit, with metadata | 151.8 MB | 123.2 MB |
| Vector, with metadata | 156.7 MB | 119.6 MB |
| Fluent Bit, without metadata | 129.5 MB | 108.9 MB |
| Vector, without metadata | 126.2 MB | 101.8 MB |
That saves 16 to 24 percent, paid for in CPU when segments are written and merged. The codec is set when an index is created, so it belongs in the index template your shipper writes through, and daily indices pick it up as they roll over.
After that, the biggest lever is the mapping. An index template that maps message as text alone, and turns off the keyword subfield on fields you never sort or aggregate on, removes the second copy of each line. We didn’t measure it here, because how much it saves depends on which fields you still need as keywords.
A rule of thumb
For logs with Kubernetes metadata on, default mappings and one replica, budget about 3 times your raw log volume. Leave out the metadata, set best_compression, and it comes down to about 2 times.
OpenSearch also stops placing new shards on a node once its disk is 85 percent full, its default low watermark, so plan against 85 percent of the disk, not all of it.
Try it yourself
Everything here is open source: OpenSearch 3.4, Fluent Bit 4.2.3 and Vector 0.53.0 on Kubernetes. The numbers come from one log mix, and yours will differ with your own lines, so run your own logs through the same steps: ship a known number of lines, check the document count matches, force-merge, then read _cat/indices and include_segment_file_sizes. The centralized logging guide sets up Fluent Bit shipping into OpenSearch end to end.
On ClusterNest Managed OpenSearch, a cluster is priced by tier and node count, not by gigabytes ingested. The storage ratio decides how many days of logs fit on the nodes you already pay for, and the bill stays the same whichever shipper you pick.
Not sure which tier your log volume needs? Write to [email protected] with your daily volume and retention, and we’ll size it with you.