Fix database cleanup table configurations for Airflow 3 - #71990
Open
Alwaysgaurav1 wants to merge 1 commit into
Open
Fix database cleanup table configurations for Airflow 3#71990Alwaysgaurav1 wants to merge 1 commit into
Alwaysgaurav1 wants to merge 1 commit into
Conversation
Update table configurations in airflow.utils.db_cleanup to align with Airflow 3 schema changes: - In task_reschedule, drop obsolete dag_id column configuration (replaced with ti_id FK in migration 0063) and include task_reschedule in task_instance.dependent_tables. - Correct asset_event dag_id_column_name to source_dag_id. - Update callback_request table name to callback. - Remove obsolete sla_miss table config. - Fix model discovery path in test_no_models_missing and add test coverage for Airflow 3 table cleanup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Updates table configurations in
airflow.utils.db_cleanupto align with Airflow 3 database schema migrations and fixes test discovery:task_reschedule:0063),dag_id,task_id,run_id,map_index, andtry_numberwere dropped fromtask_rescheduleand replaced withti_idforeign key referencingtask_instance.id.dag_id_column_name="dag_id"which previously produced SQL errors (column base.dag_id does not exist) when runningairflow db clean --dag-id."task_reschedule"totask_instance.dependent_tablesso that dependent reschedules are cleaned prior to task instances.asset_event:dag_id_column_namefrom"dag_id"to"source_dag_id"to match the table schema.callback&sla_miss:callback_requesttocallback.sla_misstable (replaced with Deadlines in Airflow 3).Test Suite Fixes:
test_no_models_missingintest_db_cleanup.pyto dynamically locateairflow.modelsacross monorepo packages.task_reschedule(recency and cascade),asset_eventDAG ID filtering, andcallbackcleanup.Testing
uv run --project airflow-core pytest airflow-core/tests/unit/utils/test_db_cleanup.py(74 passed, 0 failed).ruff.