Support backfilling a subset of a Dag's tasks - #71978
Open
Fury0508 wants to merge 5 commits into
Open
Conversation
Operators sometimes need to re-run only part of a Dag: to rebuild a single dataset among many written by one Dag, or to exclude a historical sensor whose external data has since been pruned and can no longer succeed. Until now a backfill always covered the whole graph, forcing users to either re-run everything or restructure their Dags. A backfill can now take a task_id_pattern regular expression. The matching tasks are resolved and persisted with the backfill so the selection stays reproducible if the Dag later changes, and the scheduler restricts the run to those tasks by pruning the run's Dag to the selected subset. Dependencies on unselected upstream tasks are ignored, and run success is judged on the selected tasks alone. Omitting the pattern backfills the whole Dag as before. closes: apache#70027
Fury0508
requested review from
XD-DENG,
ashb,
bbovenzi,
bugraoz93,
choo121600,
ephraimbuddy,
guan404ming,
henry3260,
jason810496,
pierrejeambrun,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
August 22, 2026 14:44
boring-cyborg
Bot
added
area:API
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Compiling a user-supplied regular expression for task selection is a denial-of-service vector (catastrophic backtracking), flagged by CodeQL. Select tasks by plain substring containment instead, mirroring how SerializedDAG.partial_subset already resolves a string selector, so no untrusted pattern is ever compiled. Regenerate the OpenAPI spec, the UI client, and the airflow-ctl datamodels to match the updated field.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backfills can now target a subset of a Dag's tasks via a
task_id_patternregular expression, instead of always covering the whole graph.Motivating cases from the issue: rebuilding one dataset among many written by a single Dag, and excluding a historical sensor whose external data has since been pruned and can no longer succeed.
Behaviour
task_id_patternis resolved to the matching task ids at creation time and persisted on the backfill (selected_task_ids), so the selection stays reproducible even if the Dag later changes.SerializedDAG.partial_subset). Dependencies on unselected upstream tasks are ignored, and run success is judged on the selected tasks alone.Scope of this PR: model + migration + REST API + scheduler semantics + tests. CLI and UI form controls are intended as follow-ups.
closes: #70027
Was generative AI tooling used to co-author this PR?