[FEATURE]: Support gpu_2 tests in the VRAM-aware pytest scheduler

[nv-tusharma]

Feature request

Extend the VRAM-aware pytest scheduler to support multi-GPU tests, starting with tests marked gpu_2.

The scheduler should understand each test's GPU count, reserve the required devices atomically, and allow compatible gpu_1 and gpu_2 tests to share multi-GPU H100 and GB200 nodes when their declared VRAM budgets fit.

Describe the problem you're encountering

Dynamo's H100 and GB200 CI nodes expose multiple GPUs, but the current VRAM-aware runner only models a single assigned GPU per test:

• test metadata does not include the gpu_1 / gpu_2 / gpu_4 / gpu_8 requirement;
• each scheduled test receives one assigned_gpu;
• child processes are launched with a single GPU index in CUDA_VISIBLE_DEVICES.

As a result, gpu_2 tests cannot participate in the VRAM-aware parallel stage. Single-GPU and multi-GPU tests must run in separate pools/stages, leaving capacity unused and increasing CI wall time.

This surfaced while stabilizing test_router_decisions_vllm_disagg in #13485. That test now has a deterministic requested_vllm_kv_cache_bytes budget, but its gpu_2 marker still prevents it from benefiting from the existing scheduler.

Proposed behavior / acceptance criteria

• Serialize the required GPU count for collected tests from their gpu_N marker.
• Represent and schedule a test against a set of GPUs rather than one GPU.
• For gpu_2, atomically reserve two eligible devices and release both when the test completes, fails, skips, or retries.
• Set CUDA_VISIBLE_DEVICES to the reserved device set in a stable order.
• Continue enforcing per-GPU VRAM budgets, multi-process safety margins, deterministic KV-cache overrides, and launch staggering where applicable.
• Define and document how profiled_vram_gib is interpreted for multi-GPU tests (for example, the maximum per-device peak), including how asymmetric usage is handled.
• Allow safe co-scheduling of gpu_1 and gpu_2 tests on the same multi-GPU node without overlapping device reservations or exceeding VRAM budgets.
• Add scheduler tests covering allocation, insufficient-device waiting, mixed gpu_1/gpu_2 workloads, retries, and cleanup without deadlock or starvation.
• Enable at least one H100/GB200 CI job to run eligible profiled gpu_1 and gpu_2 tests through the same VRAM-aware stage; unprofiled tests should continue to fall back to the sequential stage.
• Update the GPU test documentation and examples.

Support for gpu_4 and gpu_8 can follow the same resource model, but gpu_2 is the initial scope.

Describe alternatives you've tried

Today, multi-GPU tests run in separate sequential CI jobs/stages. This is safe but does not use spare GPUs or VRAM efficiently and prevents mixing single- and multi-GPU work on H100/GB200 nodes.

Context

• Slack discussion: https://ai-dynamo.slack.com/archives/C0BFP9YMQ0G/p1787325465435079?thread_ts=1787267243.121129&cid=C0BFP9YMQ0G
• Related stabilization PR: test(router): use fixed KV cache budget for vLLM disagg #13485
• Current scheduler: tests/utils/pytest_parallel_gpu.py
• Metadata collection: tests/utils/vram_utils.py