Five log shipper defaults that lose or duplicate your logs

We sent 494,175 numbered log lines through Fluent Bit and Vector into OpenSearch and counted every one that arrived. With stock settings, one shipper lost 62 percent of a busy pod’s logs and the other lost 85 percent, both without logging a single error. Five defaults were behind it, and each one is a one-line fix.

Nobody counts the logs

A log pipeline is judged by whether logs show up. Open a dashboard, search for a request ID, see a hit, move on. Nobody checks whether every line made it, because there is nothing to check it against: the logs are the record.

That is how a pipeline can lose most of a busy pod’s output for months. The lines that do arrive look fine. The gaps only show when someone goes looking for the one request that is not there, usually during an incident, which is the worst time to learn that the logs have holes.

So we built a test where every line can be counted.

Counting every line

A generator wrote 100 MB of JSON log lines, 494,175 of them, each carrying a sequence number:

{"seq": 207484, "ts": "2026-05-21T13:33:27Z", "level": "INFO",
 "service": "billing", "msg": "cache miss", ...}

After each run we pulled every sequence number out of OpenSearch with a composite aggregation and compared it with the full range. A number that never shows up is a lost line. A number that shows up twice is a duplicate. Counting documents alone would hide both whenever they cancel out.

The setup:

OpenSearch 3.6, three data nodes, default settings.
Fluent Bit 5.0.5 and Vector 0.55.0, each in its usual Kubernetes configuration, run one at a time.
Kubelet’s default log rotation: a container’s log file rotates at 10 MiB, and five files are kept.

The generator wrote to stdout at about 1 MB a second, a busy service but not an unusual one. Then we changed one thing at a time.

A busy pod outruns the shipper

With both shippers on their defaults, the counts came back like this:

Shipper	Delivered	Lost	Errors logged
Vector, `kubernetes_logs`	187,987	306,189 (62%)	none
Fluent Bit, `tail`	75,176	419,000 (85%)	none

The missing sequence numbers were not scattered. They came out as a few long blocks:

Shipper	Missing lines
Vector	0 to 207,483 and 306,295 to 404,999
Fluent Bit	0 to 270,999 and 320,000 to 467,999

Both shippers missed the first 40 to 55 seconds of the pod’s life, then read for a while, then missed another 20 to 30 seconds.

Here is what happens. At 1 MB a second, kubelet rotates the container’s log every 10 seconds or so. It renames the full file, starts a new one, compresses the older rotated files, and deletes them once there are more than five. A shipper has to notice each new file while it still exists in a form it reads.

Both shippers find new files by scanning on a timer, and both timers default to 60 seconds:

Vector’s kubernetes_logs source rescans every glob_minimum_cooldown_ms, 60,000 by default. It skips compressed files, so any file that is written, rotated and compressed between two scans is never read.
Fluent Bit’s tail input rescans every Refresh_Interval, 60 seconds by default. Its usual path, /var/log/containers/*.log, only matches the file a pod is writing now, so a file that has been rotated away before the next scan is never read either.

Neither shipper can know what it missed, which is why neither logs anything.

The official Helm charts for both keep these defaults. Fluent Bit’s default tail input sets no Refresh_Interval, and Vector’s default agent configuration is a bare kubernetes_logs source.

The fix is to scan faster than the busiest pod rotates:

sources:
  logs:
    type: kubernetes_logs
    glob_minimum_cooldown_ms: 1000

[INPUT]
    Name              tail
    Refresh_Interval  2

With those settings, both shippers delivered all 494,175 lines with no duplicates. A scan every second or two costs very little on a node, and the alternative is an outage-shaped hole in the logs of exactly the pods that log the most.

Fluent Bit repeats a backlog

A shipper also has to catch up: after a restart, after an OpenSearch outage, or when it starts on a node with logs already waiting. To test that, we wrote the same 100 MB to a file first, then pointed Fluent Bit at it.

Fluent Bit never finished. After 12 minutes it had indexed 1,988,476 documents and was still going. The exact count showed what it had been doing:

Lines	Result
0 to 218,345	indexed about 9 times each, 1,770,130 extra documents
218,346 to 494,174	never indexed

Its log repeated the same three lines, 187 times:

[ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[ warn] [output:opensearch:opensearch.0] http_do=-1 URI=/_bulk
[ warn] [engine] failed to flush chunk '1-1791383054.555840279.flb', retry in 8 seconds

OpenSearch answers a bulk request with one result per document, so a big chunk gets a big response. Fluent Bit’s opensearch output reads that response into a buffer capped at 512 KB. When a response outgrew it, Fluent Bit treated the request as failed and sent the whole chunk again, but OpenSearch had already indexed every document in it. The same chunks went round and round, and the rest of the file waited behind them.

At 1 MB a second the chunks stayed small and never hit the cap. A backlog fills chunks to the brim, and catching up after an outage is exactly when you need the logs most.

[OUTPUT]
    Name         opensearch
    Buffer_Size  False

Buffer_Size False removes the cap. The same file then arrived exactly, 494,175 lines and no duplicates, even though OpenSearch pushed back three times along the way with HTTP 429 and Fluent Bit paused reading while its memory buffer was full. Back-pressure that is handled looks like that: slower, and complete.

When OpenSearch says no to part of a request

That back-pressure was a whole request turned away. Both shippers retry that case correctly, since nothing in the request was written.

The harder case is a bulk request that half succeeds. A bulk request is split by shard, and each shard’s part goes to the node that holds it. If one node is overloaded and the others are not, OpenSearch accepts part of the request and rejects the rest, and the response is an HTTP 200 with "errors": true and a 429 on each rejected document.

To make that happen on purpose, the test index got three primary shards, one per data node, and extra writers loaded one node with a separate index pinned to it. Requests to the test index then landed partly on a busy node and partly on idle ones.

Fluent Bit retries what already worked

Fluent Bit’s opensearch output retries a chunk as a unit. Under partial rejections, that meant every document that had been accepted the first time was sent again:

[ info] [engine] flush chunk '1-1791386276.563941958.flb' succeeded at retry 2

Nothing was lost, and 76,558 lines were indexed twice. Each copy got its own generated ID, so OpenSearch had no way to tell they were the same line.

Generate_ID On makes Fluent Bit derive each document’s ID from its content. A retried document then arrives with the ID it already has, OpenSearch refuses it as already existing, and the line stays single:

[OUTPUT]
    Name         opensearch
    Generate_ID  On

Under the same load, with partial rejections in the responses, that run delivered all 494,175 lines once each.

Vector drops what was rejected

Vector does the opposite. When a response contains rejected documents, it does not retry them by default:

ERROR vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_200"
ERROR vector::sinks::util::retries: Not retriable; dropping the request. reason="error type: rejected_execution_exception, ...
ERROR vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=12858

That run lost 8,615 lines, in 5,715 small gaps: one or two lines here and there, wherever a document had landed on the busy shard. It took our heaviest load to get partial rejections into Vector’s requests, and six seconds after these lines one of the data nodes ran out of memory and restarted. The drops came first, and Vector’s log names the rejected documents as the reason. At lighter loads OpenSearch only ever turned away whole requests, which Vector retried without losing anything.

Vector’s elasticsearch sink documents the setting as request_retry_partial, off by default, and recommends pairing it with id_key so the retried documents do not become duplicates. Vector has no content-based ID of its own, so a one-line remap makes one:

transforms:
  ids:
    type: remap
    inputs:
      - logs
    source: .event_id = sha2(encode_json(.))
sinks:
  opensearch:
    type: elasticsearch
    inputs:
      - ids
    request_retry_partial: true
    id_key: event_id

With that, a partial rejection showed up in the log as a retry instead of a drop:

ERROR vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_200"
 WARN vector::sinks::util::retries: OK/retrying partial after response.

All 494,175 lines arrived, each once.

The checklist

Shipper	Setting	Default	Set it to	Without it
Vector	`glob_minimum_cooldown_ms`	60000	1000	busy pods lose lines silently
Fluent Bit	`Refresh_Interval`	60	2	busy pods lose lines silently
Fluent Bit	`Buffer_Size`	512k	`False`	catch-up repeats chunks and stalls
Fluent Bit	`Generate_ID`	`Off`	`On`	partial rejections duplicate lines
Vector	`request_retry_partial` and `id_key`	off, unset	`true`, a content hash	partial rejections drop lines

None of them costs much: a faster file scan, a little more memory for responses, and a hash per document. Content-based IDs do make OpenSearch check each ID on write, which is slower than letting it generate them, and it is still the cheaper side of the trade.

Try it yourself

Everything here is open source: OpenSearch 3.6, Fluent Bit 5.0.5 and Vector 0.55.0 on Kubernetes. The test is easy to repeat on your own pipeline. Put a sequence number in every line a test pod writes, log fast enough to rotate the file within a minute, and count what arrives with a composite aggregation over the sequence field. The centralized logging guide sets up Fluent Bit shipping into OpenSearch end to end, and the settings above drop straight into its configuration.

On ClusterNest Managed OpenSearch, a cluster is priced by tier and node count. Partial rejections are a node running out of room to index, so when they start showing up in your shipper’s logs, the answer on our side is more nodes or a larger tier, at a price you know before you change it.

Shipping logs through something we haven’t covered here? Write to [email protected] and we’ll run it through the same test.