practices guide 12 Oct 2026

Four load generators, one stalling server, and a p99 that differs by 302x

The same server, stalling for a full second every ten. One tool reported a 3 millisecond p99. Another, measuring the same stalls, reported 907 milliseconds.

By Purvansh Parmar Checked 2026-10-21 Read 8 min Entries 4

Here is a server with an obvious problem. Every ten seconds it takes a global lock and holds it for a full second, so everything in flight blocks adn everything arriving queues behind it. Any user hitting it during that second waits up to a second. Now measure it. One popular tool will tell you the p99 is three milliseconds and the server is healthy, and it will not be lying, because it genuinely never observed a slow request often enough for the percentile to notice. The reason is the thing worth understanding, and it is not about the server at all.

How this was checkedBuilt a server that takes a global lock for one second out of every ten, which is the shape of a garbage collection pause, and pointed four load generators at it. The two that hold a fixed number of connections reported p99s of 3 and 43 milliseconds. The two that send at a fixed arrival rate reported 906 and 907. Same server, same stalls, same thirty seconds, and a number that differs by a factor of 302 depending on which tool was holding the stopwatch.

What I compared on

the p99 each tool reports for the identical server and the identical stalls
whether the tool keeps sending during a stall or waits it out
whether the tool tells you it failed to keep up

01 vegeta

Open loop generator driven by a target rate, which keeps sending at that rate regardless of what the server is doing.

Wins

It measures the thing you care about. Because it keeps dispatching during the stall, roughly four hundred requests land inside each one, which is enough of the sample for the percentile to reflect it.

Loses

Its p50 of 41 ms is well above the 0.23 ms the other open loop tool measured for the same server, so its latency figure carries some of its own scheduling. Use it for the tail, not for the floor.

p99 907 ms400 req/s fixed399.47 achieved

02 k6

Scriptable generator which, with the constant-arrival-rate executor, starts iterations on a schedule rather than on completion.

Wins

The dropped iterations counter is the most useful number produced by any tool in this test. It is an explicit statement that the generator itself fell behind, which is exactly the fact a closed loop tool silently hides.

Loses

It needs a script and an executor choice to behave this way: the default executors are closed loop, so an engineer following the quickstart gets the misleading number and no warning that they chose a model.

p99 906 ms268 dropped iterationsp50 0.23 ms

03 autocannon

Node based closed loop generator, the usual choice for benchmarking a JavaScript service from the same ecosystem.

Wins

It reports percentiles out to p99.9 and p99.99 by default, which is the only reason the stall is visible in its output at all. A tool that stopped at p99 would have shown nothing.

Loses

At p99, the number almost everyone actually quotes and alerts on, it reports 43 ms for a server that freezes for a full second every ten. The pause is in the data but below the line anyone reads.

p99 43 msp99.9 1,033 ms217 req/s

04 ApacheBench

The ab binary already on most machines, holding a fixed concurrency and racing as fast as the server will answer.

Wins

It produced by far the highest throughput number in the test, and for a server with no pauses that figure is genuinely useful and takes one command with no script.

Loses

Its p99 of 3 ms is 302x lower than the open loop measurement of the same stalls. The faster it goes, the more it dilutes the pause, so the healthier a stalling server looks the harder you push it.

p99 3 ms302x lower than vegetamax 2,296 ms

Side by side

	Model	p50	p99	Max
ApacheBench	closed loop	1 ms	3 ms	2,296 ms
autocannon	closed loop	41 ms	43 ms	1,036 ms
k6 constant arrival	open loop	0.23 ms	906 ms	1,000 ms
vegeta	open loop	41 ms	907 ms	1,043 ms

Verdict

Use k6 with the constant-arrival-rate executor, and read the dropped iterations line as carefully as you read the percentiles. It was the only tool that both measured the stall properly and told me it had failed to keep up while doing so, which together are the two halves of the answer. The general rule this test makes concrete: if your generator waits for a response before sending the next request, it stops sampling precisely when the server is worst, so its tail latency describes the server's good moments at high resolution and its bad ones barely at all. Check which model your tool uses before you trust a p99 from it.

Sources

PreviousA hundred thousand vectors, six HNSW settings, and the one that beat the biggest NextOne SQLite database, seven durability settings, and the 149x nobody turns on