[Pooling] Report input throughput for batched requests

[taneem-ibrahim]

Purpose

vllm bench serve reports pooling throughput per HTTP request. With batched embedding or reranking inputs, one request can represent several independently processed inputs, so request throughput alone understates useful model throughput and makes batch-size comparisons difficult.

The root cause is that pooling metrics increment completed once per successful HTTP response and discard the request's logical input cardinality. This change records that cardinality before sending the request and adds two pooling only metrics:

• total_input_sequences
• input_sequence_throughput

Existing request and token metrics are unchanged. Embedding and generic pooling requests count batched inputs; reranking requests count documents. The Python and Rust benchmark clients expose the same console and JSON metrics.

Reproducer

Ten embedding inputs batched as [4, 4, 2] produce three successful HTTP
requests over a one-second measurement window.

Output on main

{'http_requests': 3, 'batch_cardinalities': [4, 4, 2], 'source_inputs': 10, 'reported_request_throughput': 3.0}

main exposes the transport rate but not the ten-input processing rate.

Output on this branch

{'http_requests': 3, 'batch_cardinalities': [4, 4, 2], 'source_inputs': 10, 'reported_request_throughput': 3.0, 'reported_input_sequence_throughput': 10.0}

The existing 3.0 req/s result is preserved and the benchmark now also reports
10.0 inputs/s.

Test plan and results

.venv/bin/python -m pytest \
  tests/benchmarks/test_random_dataset.py \
  tests/benchmarks/test_rust_bench_cli_parity.py -q

13 passed, 42 warnings in 3.24s

cd rust
cargo test -p vllm-bench

154 passed, 0 failed, 10 ignored
3 CLI-parity tests passed, 0 failed

cd rust
cargo fmt --all -- --check

Passed

A local aiohttp smoke test exercised the embeddings, generic pooling, and reranking request paths against a mock server:

{'http_requests': 3, 'input_sequences': 9, 'request_throughput': 3.0, 'input_sequence_throughput': 9.0}

End-to-end validation

Validated on an NVIDIA B300 using BAAI/bge-base-en-v1.5.

Client/path	Batch size	HTTP requests	Input sequences	Result
Python embeddings	1	10	10	Passed
Python embeddings	4	3	10	Passed
Python pooling	4	3	10	Passed
Rust embeddings	4	3	10	Passed

For the non-divisible batch, ten inputs were correctly represented as three HTTP requests (4 + 4 + 2). Existing request and token metrics were preserved, and input_sequence_throughput matched total_input_sequences / benchmark_duration.

AI assistance disclosure

OpenAI Codex (GPT-5) assisted with drafting the implementation and root cause analysis. Before submitting this PR, I reviewed every changed line and independently ran the test commands listed above.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[yewentao256]

Thanks for the work!

[yewentao256]

When prompt_list is None, documents can still come from extra_body, but this always reports one input even if multiple documents are sent?

[yewentao256]

Should we also expose this metric in the Rust compare, multi-run, and sweep summaries?

[yewentao256]

Will list[list[int]] be supported?

[taneem-ibrahim]

Very good point. The current helper treats list[list[int]] as one input. I’ll count both list[str] and list[list[int]] as batches while preserving list[int] as a single input, and update the type annotations accordingly.

Reproducer to test:

.vent/bin/python - <<'PY'
from vllm.benchmarks.lib.endpoint_request_func import _get_num_input_sequences

cases = [
    ("text batch", ["a", "b"], 2),
    ("token-ID batch", [[101, 102], [103, 104]], 2),
    ("single token-ID input", [101, 102], 1),
    ("chat content", [{"type": "text", "text": "hello"}], 1),
]

for name, value, expected in cases:
    actual = _get_num_input_sequences(value)
    print(f"{name}: expected={expected}, actual={actual}")
    assert actual == expected

print("All input-shape checks passed")
PY

Prior to the latest commit, we get an AssertionError:

text batch: expected=2, actual=2
token-ID batch: expected=2, actual=1
Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
AssertionError

After making the changes, it passes:

text batch: expected=2, actual=2
token-ID batch: expected=2, actual=2
single token-ID input: expected=1, actual=1
chat content: expected=1, actual=1
All input-shape checks passed

Thanks for the review!

[yewentao256]

LGTM, thanks for the work!

[github-actions]

✅ @taneem-ibrahim, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[taneem-ibrahim]

/ci run

[github-actions]

✅ Triggered Buildkite CI #85126 for commit 80e3fefa6c54.

[taneem-ibrahim]

/ci retry

[github-actions]

✅ Queued 1 failed job(s) for retry in Buildkite CI #85126.