[Bugfix][KV Offload] Decouple shared-region creator ownership - #53073
Open
Alex-ai-future wants to merge 2 commits into
Open
[Bugfix][KV Offload] Decouple shared-region creator ownership#53073Alex-ai-future wants to merge 2 commits into
Alex-ai-future wants to merge 2 commits into
Conversation
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Alex <jihui.huang@daocloud.io>
Contributor
Author
|
@Etelis @orozery PTAL |
Signed-off-by: Alex <jihui.huang@daocloud.io> Co-authored-by: Codex <noreply@openai.com>
Alex-ai-future
force-pushed
the
fix/kv-offload-shared-region-owner
branch
from
August 20, 2026 08:30
e58704e to
b445ce7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR separates the transient shared-region initializer from the lifetime
owner of singleton operations.
O_CREAT | O_EXCLelects a process only for the initialization branch thatchecks
/dev/shmcapacity and callsftruncate(). Calling that process the"creator" is useful within that branch, but it is not a durable ownership
concept: it is an incidental result of startup timing, may belong to a process
that has already exited, and says nothing about which live worker should run a
later one-time action.
This becomes a concrete failure mode when the initializer exits after the file
has reached the expected size but before the other workers finish their
lifetime. The remaining workers can open and map the region successfully, yet
every live object may be a joiner with
_creator=False. Any later singletonoperation gated on
_creator, including cleanup unlink, then has no executorat all and can leave the
/dev/shmpath behind. The same structural problemapplies whenever a future one-time operation is tied to this historical,
unobservable initializer identity.
Persisting
_creatortherefore gives cleanup and future singleton operationsan unnecessary and unreliable restriction. This PR removes that lifecycle
state and makes
SharedOffloadRegionresolve singleton ownership from theactual node-local rank, so a live, deterministic rank-0 worker can perform
singleton actions regardless of which process won
O_EXCL.The existing
O_EXCL/ftruncate()/ joiner polling protocol is intentionallyunchanged. In particular, this PR does not address creator-crash recovery,
stale-file takeover, orphan sweeping, or map-after-barrier unlink timing.
This is not a duplicate of #51317 or #52596: those PRs change the shared-memory
cleanup protocol, but both retain
_creatoras the post-initialization unlinkgate. This PR only replaces that ownership model.
AI assistance was used to prepare this change. The human submitter must review
every changed line and be able to explain the initialization and cleanup
semantics before submitting.
Test Plan
O_EXCLinitialization and joiner waiting remain unchanged.selected by local rank to perform singleton cleanup.
O_EXCLwinner,determines singleton cleanup.
Test Result
Ran on Linux:
PyTorch emitted 14 deprecation warnings. There were no test failures or skips.
No model evaluation is required because this change does not affect model
outputs, accuracy, or serving behavior.