Add deferrable mode to InfluxDB3Operator - #71976
Open
subhramit wants to merge 4 commits into
Open
Conversation
Signed-off-by: subhramit <subhramit.bb@live.in>
Signed-off-by: subhramit <subhramit.bb@live.in>
Signed-off-by: subhramit <subhramit.bb@live.in>
InfluxDB3Operator
Author
|
Ah, I think I missed a lowest-direct-dependencies edge case here. The spellcheck also fails, will address both in the next commit. Update - fixed. |
Signed-off-by: Subhramit Basu <subhramit.bb@live.in>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #67107
Add deferrable support to
InfluxDB3OperatorWhen
deferrable=True, the operator now defers to a newInfluxDB3QueryTriggerso the worker slot is released while the query runs. The trigger awaitsInfluxDBClient3.query_async()and resumes the task with the same JSON-serializable record shape returned by the synchronous path.Also update provider metadata and documentation to surface the trigger, add a deferrable example, and bump the
influxdb3-pythonminimum version to>=0.12.0(the release that introducedquery_async()).Note that this deviates from the issue's proposed API as I did not add a "
poll_interval"/polling loop. InfluxDB 3'squery_asyncstreams the full result over one Arrow Flight call, which means there's no server-side job to poll a status on, unlike Snowflake/BigQuery/Redshift. The trigger awaits the query once and emits a single event.This is more similar in shape to
SQLExecuteQueryTrigger(used byGenericTransfer's deferrable path) - which also has a single await yielding oneTriggerEventwith nopoll_intervalfor the same reason:airflow/providers/common/sql/src/airflow/providers/common/sql/triggers/sql.py
Lines 92 to 102 in 4e4d060
Have added tests for the new deferrable operator path, trigger serialization/execution, and async hook behavior.
Was generative AI tooling used to co-author this PR?
Generated-by: GPT-5.4 following the guidelines
All code changes (and also this description) were manually touched up and reviewed by me.