Skip to content

Add deferrable mode to InfluxDB3Operator - #71976

Open
subhramit wants to merge 4 commits into
apache:mainfrom
subhramit:deferrable-influxdb3-operator
Open

Add deferrable mode to InfluxDB3Operator#71976
subhramit wants to merge 4 commits into
apache:mainfrom
subhramit:deferrable-influxdb3-operator

Conversation

@subhramit

@subhramit subhramit commented Aug 22, 2026

Copy link
Copy Markdown

Closes #67107

Add deferrable support to InfluxDB3Operator

When deferrable=True, the operator now defers to a new InfluxDB3QueryTrigger so the worker slot is released while the query runs. The trigger awaits InfluxDBClient3.query_async() and resumes the task with the same JSON-serializable record shape returned by the synchronous path.
Also update provider metadata and documentation to surface the trigger, add a deferrable example, and bump the influxdb3-python minimum version to >=0.12.0 (the release that introduced query_async()).

Note that this deviates from the issue's proposed API as I did not add a "poll_interval"/polling loop. InfluxDB 3's query_async streams the full result over one Arrow Flight call, which means there's no server-side job to poll a status on, unlike Snowflake/BigQuery/Redshift. The trigger awaits the query once and emits a single event.
This is more similar in shape to SQLExecuteQueryTrigger (used by GenericTransfer's deferrable path) - which also has a single await yielding one TriggerEvent with no poll_interval for the same reason:

async def run(self) -> AsyncIterator[TriggerEvent]:
try:
self.log.info("Extracting data from %s", self.conn_id)
self.log.info("Executing: \n %s", self.sql)
self.log.info("Reading records from %s", self.conn_id)
results = await self._get_records()
self.log.info("Reading records from %s done!", self.conn_id)
self.log.debug("results: %s", results)
yield TriggerEvent({"status": "success", "results": results})

Have added tests for the new deferrable operator path, trigger serialization/execution, and async hook behavior.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: GPT-5.4 following the guidelines
All code changes (and also this description) were manually touched up and reviewed by me.

Signed-off-by: subhramit <subhramit.bb@live.in>
Signed-off-by: subhramit <subhramit.bb@live.in>
Signed-off-by: subhramit <subhramit.bb@live.in>
@subhramit subhramit changed the title Add deferrable mode to InfluxDB3Operator Add deferrable mode to InfluxDB3Operator Aug 22, 2026
@subhramit

subhramit commented Aug 22, 2026

Copy link
Copy Markdown
Author

Ah, I think I missed a lowest-direct-dependencies edge case here. The spellcheck also fails, will address both in the next commit.

Update - fixed.

Signed-off-by: Subhramit Basu <subhramit.bb@live.in>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

InfluxDB3Operator: add deferrable variant

1 participant