[Model Runner V2] Reserve CUDA graph memory - #53306
Open
njhill wants to merge 8 commits into
Open
Conversation
Model Runner V2 became the default for dense models in 0.25.1, but its profile_cudagraph_memory() was a placeholder returning 0. Worker. determine_available_memory() subtracts this estimate before sizing the KV cache, so with V2 no headroom was reserved for CUDA graph capture: the KV cache claimed the whole gpu_memory_utilization budget and capture_model() OOMed at startup (e.g. Llama-3.1-70B FP8, TP=8 on L40S). Implement profile_cudagraph_memory() for the V2 runner, reusing its own initialize_kv_cache()/capture_model(): bootstrap a minimal KV cache, capture graphs into a throwaway pool (so their memory is reclaimed and does not pollute the persistent global pool), measure the free-memory delta, then release the profiling state while keeping model weights. Fixes vllm-project#49224 Signed-off-by: Anh Tran <anh.tran.3889@gmail.com>
…ph-memory-profiling
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
4 tasks
Member
Author
|
/ci run all |
|
✅ Triggered Buildkite CI #85073 for commit |
…ph-memory-profiling
Member
Author
|
/ci run all |
|
✅ Triggered Buildkite CI #85113 for commit |
WoosukKwon
approved these changes
Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profile cuda graph memory usage at startup so that it can be factored into the kv cache auto-sizing, as MRV1 already does.
Note we are hoping to replace this with #50779 soon but it will unblock the change to make MRV2 default for all models in the meantime.
This includes the commits from #49233 by @anhtra3889 plus some fixes / additional rework.
Fixes: #49224