Skip to content

Fix database cleanup table configurations for Airflow 3 - #71990

Open
Alwaysgaurav1 wants to merge 1 commit into
apache:mainfrom
Alwaysgaurav1:fix-db-clean-airflow3-tables
Open

Fix database cleanup table configurations for Airflow 3#71990
Alwaysgaurav1 wants to merge 1 commit into
apache:mainfrom
Alwaysgaurav1:fix-db-clean-airflow3-tables

Conversation

@Alwaysgaurav1

Copy link
Copy Markdown

Description

Updates table configurations in airflow.utils.db_cleanup to align with Airflow 3 database schema migrations and fixes test discovery:

  1. task_reschedule:

    • In Airflow 3 (migration 0063), dag_id, task_id, run_id, map_index, and try_number were dropped from task_reschedule and replaced with ti_id foreign key referencing task_instance.id.
    • Removed obsolete dag_id_column_name="dag_id" which previously produced SQL errors (column base.dag_id does not exist) when running airflow db clean --dag-id.
    • Added "task_reschedule" to task_instance.dependent_tables so that dependent reschedules are cleaned prior to task instances.
  2. asset_event:

    • Corrected dag_id_column_name from "dag_id" to "source_dag_id" to match the table schema.
  3. callback & sla_miss:

    • Updated table name from callback_request to callback.
    • Removed obsolete sla_miss table (replaced with Deadlines in Airflow 3).
  4. Test Suite Fixes:

    • Fixed test_no_models_missing in test_db_cleanup.py to dynamically locate airflow.models across monorepo packages.
    • Updated exclusion list to accurately account for all 52 Airflow 3 declarative models.
    • Added unit tests covering task_reschedule (recency and cascade), asset_event DAG ID filtering, and callback cleanup.

Testing

  • Ran unit tests: uv run --project airflow-core pytest airflow-core/tests/unit/utils/test_db_cleanup.py (74 passed, 0 failed).
  • Formatted and linted with ruff.

Update table configurations in airflow.utils.db_cleanup to align with Airflow 3 schema changes:
- In task_reschedule, drop obsolete dag_id column configuration (replaced with ti_id FK in migration 0063) and include task_reschedule in task_instance.dependent_tables.
- Correct asset_event dag_id_column_name to source_dag_id.
- Update callback_request table name to callback.
- Remove obsolete sla_miss table config.
- Fix model discovery path in test_no_models_missing and add test coverage for Airflow 3 table cleanup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant