Skip to content

Keep MSGraph pagination offset across paginated pages - #71986

Draft
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:msgraph-pagination-keyerror
Draft

Keep MSGraph pagination offset across paginated pages#71986
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:msgraph-pagination-keyerror

Conversation

@FrankYang0529

Copy link
Copy Markdown
Member

Why

  • default_pagination() recovered the $skip offset with sum(len(result["value"]) for result in responses()).
  • A user can define result_processor as lambda result, **context: result.get("value"). append_result() then extends the accumulator with the page's items, so XCom holds bare messages and those have no "value" key. On the deferrable path responses() reads exactly that XCom, so the run dies with KeyError: 'value' once a second full page arrives.

How

  • Derive $skip from the query parameters the page was requested with, so the offset no longer depends on what a user callback left in XCom. The same expression fixes the off-by-one page in paginated_run() and keeps a user-supplied starting offset.
  • Carry those query parameters across the defer boundary with defer(kwargs=...) and restore them in execute_complete().

Verification

  • uv run --project providers/microsoft/azure --frozen pytest providers/microsoft/azure/tests/unit -q

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Signed-off-by: PoAn Yang <payang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant