[BUG]: cancelled ThunderAgent admission leaves a phantom paused program

[MatejKosec]

Description

Cancelling a ThunderAgent request while before_request is waiting for admission leaves the request's program state in the scheduler.

_admit_locked calls ProgramTable.begin_request before capacity is available. For a new program, that inserts it in programs; admission then marks it PAUSED and inserts it in paused. before_request handles only asyncio.TimeoutError, so an asyncio.CancelledError from client disconnect or request cancellation propagates without rolling back those mutations.

The request handler awaits before_request before entering the try/finally that calls after_request, so this cancellation has no later cleanup path. The abandoned entry triggers the fairness rule that queues unrelated new programs behind any paused program. A later scheduler tick can also resume and assign the abandoned program, accounting capacity for a request that no longer exists.

Repeated cancellations with distinct program IDs can therefore grow the program table, block subsequent admissions, and create phantom worker assignments.

Source

At 6e80bf9324eceb9d60856e90308eba6621ea42da:

• begin_request inserts a new program and mutates its step and status.
• before_request catches timeout but not cancellation.
• _admit_locked inserts the program before admission and records a failed admission in paused.
• The new-program fairness rule treats any retained paused entry as a reason to queue, and the scheduler later resumes paused entries without knowing whether their waiter was cancelled.
• The request handler enters its cleanup try/finally only after before_request returns.
• An existing test cancels an admission waiter but does not assert scheduler state afterward.

Expected behavior

Admission cancellation should atomically undo the mutations made for that admission attempt. A newly created program should not remain in either table. For an existing program, cancellation should restore its pre-request state rather than deleting valid session history. Cleanup should be race-safe if a scheduler tick resumes the program while cancellation is being delivered.

Regression coverage

Fill capacity, start admission for a fresh program, wait until it is present in paused, cancel the task, and verify that the program is absent from both programs and paused. A subsequent scheduler tick must not assign a worker or consume capacity for it, and another new program must not be held by the fairness rule. Add a separate existing-program case that verifies restoration of its pre-request lifecycle, status, step count, token count, assignment, and waiter state.

Duplicate and overlap check

Current issue and pull-request searches for ThunderAgent admission cancellation, disconnects, CancelledError, paused-state leaks, and phantom programs found no report or fix for this behavior. Open PR #12207 changes hybrid-KV capacity accounting and paused-program placement; PR #12990 only reformats a metric expression. Neither adds cancellation rollback.