Skip to content

Honor execute_tasks_new_python_interpreter when the task supervisor forks the task runner - #71995

Open
goingforstudying-ctrl wants to merge 2 commits into
apache:mainfrom
goingforstudying-ctrl:fix/task-sdk-honor-new-interpreter-config
Open

Honor execute_tasks_new_python_interpreter when the task supervisor forks the task runner#71995
goingforstudying-ctrl wants to merge 2 commits into
apache:mainfrom
goingforstudying-ctrl:fix/task-sdk-honor-new-interpreter-config

Conversation

@goingforstudying-ctrl

Copy link
Copy Markdown
Contributor

Been digging into #71707 — the case where a task subprocess hangs forever with every thread stuck at ssl.py:440 building fresh SSLContexts. The mechanism there is that ActivitySubprocess.start() does a bare os.fork() on Linux, and if a sibling thread in the supervisor (OTel exporter, google-auth refresh threads, an OpenLineage listener...) happens to be inside OpenSSL's global lock at that exact moment, the child inherits the held lock with no owner left to release it. Every later SSLContext construction in that child then blocks forever.

What surprised me is that [core] execute_tasks_new_python_interpreter=True doesn't actually protect you from this. The executor honors the knob when launching the supervisor (e.g. celery's _execute_in_subprocess), but the supervisor then bare-forks the task runner regardless — the config stops one fork short of the process that actually runs user code. The Edge worker hit the same class of problem and got the config honored for its launch path in #65943; this does the equivalent for the Task SDK's own fork.

The fork+exec machinery itself already exists and isn't macOS-specific (it's what _should_use_exec() gates), so the change is small: when the config is set and we're launching the real task entry point, take the exec path too. Test stub targets keep bare fork as before, and the DAG processor / triggerer paths are untouched since the option is task-scoped.

Verified on Linux: with the config set the task child comes up as a fresh interpreter (_child_exec_main rehydrating over the dup2'd FDs, supervisor comms working end to end), ruff/mypy are clean, and the existing supervisor test suite passes (the handful of test_exit_by_signal/remote-logging errors fail identically on a clean checkout — missing optional deps locally, unrelated). Added selection tests for default / config-set / fork-unsafe-platform / stub-target combinations.

closes: #71707


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

The config currently stops one fork short: the executor honors it when
launching the supervisor, but the Task SDK supervisor then bare-forks the
task runner anyway. On a long-running multithreaded supervisor (OTel
exporter, google-auth refresh threads, OpenLineage listener) that fork
can inherit OpenSSL's global lock mid-held by a sibling thread, and every
SSLContext construction in the child then blocks forever (issue apache#71707).

Reuse the existing fork+exec machinery (already used on macOS) so Linux
deployments can opt into a fresh interpreter per task with the same
[core] execute_tasks_new_python_interpreter knob, matching what the Edge
worker did in apache#65943.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor

1 participant