Honor execute_tasks_new_python_interpreter when the task supervisor forks the task runner - #71995
Open
goingforstudying-ctrl wants to merge 2 commits into
Conversation
The config currently stops one fork short: the executor honors it when launching the supervisor, but the Task SDK supervisor then bare-forks the task runner anyway. On a long-running multithreaded supervisor (OTel exporter, google-auth refresh threads, OpenLineage listener) that fork can inherit OpenSSL's global lock mid-held by a sibling thread, and every SSLContext construction in the child then blocks forever (issue apache#71707). Reuse the existing fork+exec machinery (already used on macOS) so Linux deployments can opt into a fresh interpreter per task with the same [core] execute_tasks_new_python_interpreter knob, matching what the Edge worker did in apache#65943.
goingforstudying-ctrl
requested review from
amoghrajesh,
ashb and
kaxil
as code owners
August 23, 2026 06:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Been digging into #71707 — the case where a task subprocess hangs forever with every thread stuck at
ssl.py:440building freshSSLContexts. The mechanism there is thatActivitySubprocess.start()does a bareos.fork()on Linux, and if a sibling thread in the supervisor (OTel exporter, google-auth refresh threads, an OpenLineage listener...) happens to be inside OpenSSL's global lock at that exact moment, the child inherits the held lock with no owner left to release it. Every laterSSLContextconstruction in that child then blocks forever.What surprised me is that
[core] execute_tasks_new_python_interpreter=Truedoesn't actually protect you from this. The executor honors the knob when launching the supervisor (e.g. celery's_execute_in_subprocess), but the supervisor then bare-forks the task runner regardless — the config stops one fork short of the process that actually runs user code. The Edge worker hit the same class of problem and got the config honored for its launch path in #65943; this does the equivalent for the Task SDK's own fork.The fork+exec machinery itself already exists and isn't macOS-specific (it's what
_should_use_exec()gates), so the change is small: when the config is set and we're launching the real task entry point, take the exec path too. Test stub targets keep bare fork as before, and the DAG processor / triggerer paths are untouched since the option is task-scoped.Verified on Linux: with the config set the task child comes up as a fresh interpreter (
_child_exec_mainrehydrating over the dup2'd FDs, supervisor comms working end to end),ruff/mypyare clean, and the existing supervisor test suite passes (the handful oftest_exit_by_signal/remote-logging errors fail identically on a clean checkout — missing optional deps locally, unrelated). Added selection tests for default / config-set / fork-unsafe-platform / stub-target combinations.closes: #71707
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.