Why is disabled Keep-alive ???

[joanhey]

I didn't know why the results were so bad in the last results !!!

Later I read that keep-alive is disabled !!! 😕

Anybody can send me to the issue that explain this change ?

If we don't use keep-alive, that's only necessary in the ancient H1.0, because by default is enabled in H1.1, H2 and H3. And we are disabling the default 😕 .
If we test without keep-alive we are not testing the framework performance, we are testing the kernel performance.

I can explain more, but I prefer to listen the reasons for this change.

Thank you.

PD: https://en.wikipedia.org/wiki/HTTP_persistent_connection

[waghanza]

Hi @joanhey ,

Thanks for this question. In fact, this has been a debate for long, thier is a lot of issues about that here.

My first of all, the purpose of this project is to see how framework could react in production (how many user could be handled by X or Y framework).

How keep alive could help us in this purpose ?

Regards,

[eastspire]

Hi @waghanza,

Thanks for engaging on this. I'd like to gently add another +1 to the keep-alive discussion.

A few observations that I think make this worth revisiting:

1. Default behavior matters. As @joanhey noted, persistent connections are on by default in HTTP/1.1, HTTP/2, and HTTP/3. Most production clients (browsers, mobile apps, service-to-service RPC) reuse connections. A benchmark that explicitly disables keep-alive is therefore measuring a setup that doesn't match real-world usage.

2. What we end up measuring. When keep-alive is off, the throughput is heavily bound by kernel-level TCP setup/teardown (SYN/SYN-ACK/ACK, TIME_WAIT, port reuse) rather than the framework's request handling. Frameworks with mature async runtimes may still win, but the gap shrinks or inverts in ways that don't reflect real performance.

3. Reproducibility across runtimes. Some runtimes (Go, Java NIO) handle short-lived connections very efficiently, while others (Python WSGI, classic PHP-FPM) suffer disproportionately. Without keep-alive, the ranking reflects connection-handling efficiency more than framework efficiency.

4. The H2/H3 angle. Modern HTTP versions are connection-multiplexed by design. If the goal is to compare "how does this framework look in a modern stack", keep-alive (or better, H2) is closer to that goal.

Would you be open to publishing a short methodology note explaining the rationale for the current keep-alive setting? That would help a lot — it would close out the recurring duplicate questions (you mentioned this has been a debate for a long time), and give newcomers a single place to read the reasoning instead of having to dig through old threads.

Thanks for the great work on this project — it's a really valuable resource.

[joanhey]

My first of all, the purpose of this project is to see how framework could react in production (how many user could be handled by X or Y framework).

How any app use in production the HTTP stack, I think than the default. If you want to test like HTTP1.0 by default don't use keep-alive. A modification later for better performance was to add Kepp-Alive, that in this bech we disable it, when in HTTP1.1, 2, 3 is by default.

We are testing how fast is the kernel handshake, not the framework.

But I think that any app actually use TLS, so we need to add to the HTTP the TLS overhead for each connection 🤔 .

What are really testing this benchmark ??

[waghanza]

This project is about testing frameworks in real conditions, not stress conditions.

I mean their is 2 purposes for testing and those ways are différents.

Test the limit where you can go for one client (check vertical scaling) and test how many clients the stack could handle (horizontal scaling).

The idea is to check the second.

I'm not sure keep alive will help in that case.

Do you think this is the case ? And why ?

[joanhey]

This bench is using concurrent connections for the test: 64, 256 and 512.

So never will know how many clients could handle, as is limited.

What we want is how fast can serve this quantity of clients, at the same time.

[waghanza]

Thanks everybody to perticipate to this, maybe I got it wrong.

The idea of this project is to score framework depending on customer they could handle, higher score, higher customer.

Maybe we should :

• choose only one concurrency (calculated with hardware specs)
• use a linear rate ramp
• specify timeout
• keep a timeseries (not only final response, but this is for an other graph)

So keep alive should be re-enabled, or live it as changed @floatdrop

[waghanza]

@nigrosimone I think this might interest you as well

[nigrosimone]

Keep-alive on is the right default: it is what HTTP/1.1 does, and without it every framework compresses into the same number, the test measures connection setup in the kernel. If the worry is that keep-alive hides the accept path, a middle way is a short-lived profile: keep-alive on, but the client closes each connection after N requests (HttpArena does this with N=10). For the rest, zrk already has what you list: -R A:B is the linear ramp, --deadline the timeout, --timeseries the series, and an open-loop rate avoids coordinated omission that --closed cannot see. Happy to re-test the fulmine entry with the new settings.

[eastspire]

Following up on this thread now that @floatdrop's PR (#9714) has clarified the path forward — I think the proposed dual-mode approach is the right call and worth supporting here:

zrk 2.4.0 adds --disable-keepalive, so the collect targets use it. […] Each result file records which mode produced it in its own config.disable_keepalive.

Two points in favor, and one practical note:

1. It settles the long-standing debate without picking a side. The same complaint ("we're measuring the kernel, not the framework") shows up in nearly every keep-alive discussion. Running both modes turns the question into data instead of opinion — readers can pick the column that matches their deployment. The per-file config.disable_keepalive flag (per PR description) makes the mode self-describing, which is exactly what's been missing.

2. It surfaces the runtime asymmetry @joanhey raised. Frameworks that ignore Connection: close (the Nim httpbeast-derived ones, Workerman/Swoole PHP, agoo-c, may_minihttp, httpz, mist — listed in the PR description) currently sit at the top of the board purely because the load generator reuses sockets they never closed. With --disable-keepalive enforced from the client side, every framework pays the same TCP-setup cost and the ranking reflects request handling rather than connection-handling policy. That single change closes a ~130x spread that has nothing to do with the frameworks being compared.

3. Practical note. The PR also bumps the zrk version check in run.sh from "presence only" to sort -V-based comparison. That's the right way to do it — silent acceptance of an old zrk fails hours into a run with empty .results. Worth keeping that part of the change even if the bench-mode split is debated further.

[waghanza]

Maybe we can agreed on any command to run

zrk \
  -c256 \ # one concurrency only
  -d120s \
  -R1000:100000 \ # linear ramp up
  --interval 1s \
  --timeout 8s \
  --latency \
  --timeseries results.ndjson \ # we can show a timeseries graph later
  --hdr latency.hgrm \ # full latency capture
  --format json \
  -o result.json \
  http://127.0.0.1:3000

what do you think @floatdrop @nigrosimone @joanhey @eastspire ?

[floatdrop]

@waghanza you may want to throw in --timeseries-histogram flag, so each interval will contain full latency histogram encoded as HdrHistogram in base64 (it will increase size of results.ndjson), so you can plot high accuracy latency graphs.

[waghanza]

With the above command (with few adaptations tho), I have

language	framework	req/s for 64 concurrency	req/s for 256 concurrency	req/s for 512 concurrency
crystal	amber	49910.83	47607.24	46771.68
crystal	lucky	44756.46	45137.73	41489.02
crystal	marten	50432.27	49046.47	48072.85
csharp	carter	50475.02	50325.46	50260.44
fsharp	suave	50473.40	50342.98	50250.73
go	fiber	50471.12	50351.03	50256.33
go	goyave	50471.98	50361.06	50246.93
go	gramework	23084.85	22870.86	22890.30
javascript	elysia-bun	50471.46	50364.32	50273.08
javascript	elysia-node	49785.22	49117.67	48287.07
javascript	express	49492.77	48704.69	48260.00
javascript	express-bun	32032.75	30520.01	30549.91
javascript	express-deno	25673.06	23993.92	23856.70
javascript	hyper-express	50474.06	50329.80	50209.70
javascript	iotjs-express	50090.49	49597.03	49138.22
javascript	nestjs-express	43586.86	43472.93	43487.84
javascript	ultimate-express	50469.03	50322.72	50251.80
php	laravel	3755.31	3888.18	3857.93
php	laravel-s-laravel	14439.45	14566.97	14330.17
php	laravel-s-lumen	19747.01	20209.44	20208.53
php	symfony	5067.89	7131.28	7034.53
ruby	hanami	27698.75	32901.85	34730.26
ruby	hanami-api	41768.26	44715.38	44180.23
ruby	rage	49370.70	47539.06	46008.82
ruby	rails	15667.46	16541.29	16543.83
ruby	rails-api	17530.80	18523.64	18687.38