[BUG]: Mid-rollout DGD revert bypasses maxUnavailable and deletes serving DCDs

[tmonty12]

Describe the Bug

Reverting a DynamoGraphDeployment worker specification while a rolling update is still in progress can immediately delete the serving worker generation, bypassing the configured maxUnavailable limit.

Using A and B to represent worker-spec hashes:

Initial:
  desired=A, current=A, DCDs=[A]

Upgrade in progress:
  desired=B, current=A, phase=InProgress, DCDs=[A, B]

User reverts to A:
  desired=A, current=A, phase=InProgress, DCDs=[A, B]

The controller interprets current == desired && phase == InProgress as recovery from an interrupted rollout completion. It calls completeRollingUpdate(A), which deletes every non-A worker DCD, including generation B while B may still be serving traffic.

These deletions bypass the rolling-update replica planner, so nvidia.com/deployment-rolling-update-max-unavailable is not respected.

Steps to Reproduce

1. Create a DGD with worker specification A and multiple worker replicas.

2. Configure the rolling-update strategy with nvidia.com/deployment-rolling-update-max-unavailable.

3. Update the worker specification from A to B.

4. Wait until both A and B DCDs exist and B is serving replicas, but before .status.rollingUpdate.phase reaches Completed.

5. Reapply the original worker specification A.

6. Observe the controller log:

   Detected stuck rolling update: hashes match but phase is InProgress
   

7. Observe that the B DCDs are deleted directly rather than being scaled down within the configured availability budget.

The issue is easiest to reproduce late in the A→B rollout, when B serves most of the desired replicas but A remains recorded as the current worker hash.

Expected Behavior

Changing the desired worker hash during an active rollout should be treated as a superseding rollout, including when the new desired hash equals the previously current hash.

For A→B→A:

• A should become the new rollout target.
• B should be treated as the old serving generation.
• B replicas should be drained within the configured maxUnavailable and maxSurge constraints.
• B DCDs should only be deleted after they have been scaled to zero and their workloads have terminated.
• Aggregate availability should remain at or above desiredReplicas - maxUnavailable.

Actual Behavior

When the desired specification is reverted to A, the controller observes:

current=A, desired=A, phase=InProgress

It executes the following recovery path:

if current.contains(newWorkerHash) &&
    rollingUpdateStatus.Phase == nvidiacomv1beta1.RollingUpdatePhaseInProgress {
    return r.completeRollingUpdate(ctx, dgd, status, newWorkerHash)
}

Because newWorkerHash is A, completeRollingUpdate() calls deleteOldWorkerDCDs(A). This directly deletes every worker DCD whose hash is not A, including generation B while it may still have serving replicas.

The deletion does not pass through buildRollingUpdateContext(), so the configured maxUnavailable constraint is bypassed.

Environment

• Dynamo operator: v1.3.0
• Also present by code inspection in v1.3.1
• Still present on main as of commit fe34cb000c41075650cd6ad711074709ae472a05
• Deployment type: operator-managed, DCD-backed workers
• Update strategy: rolling update with maxUnavailable configured