Skip to content

[improvement](be) Optimize flat Variant assembly for sparse rows - #67423

Closed
eldenmoon wants to merge 1 commit into
apache:masterfrom
eldenmoon:branch-variant-v2-direct-assembler
Closed

[improvement](be) Optimize flat Variant assembly for sparse rows#67423
eldenmoon wants to merge 1 commit into
apache:masterfrom
eldenmoon:branch-variant-v2-direct-assembler

Conversation

@eldenmoon

@eldenmoon eldenmoon commented Sep 2, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: VariantAssembler scanned every flat materialized path for every row even when only a small fraction of paths were present. This change builds a batch-local active-slot CSR index only for the narrow whole-root, one-part, non-array, non-conflicting layout with empty root payloads. The fast path is used only when at most 1/16 of row/path cells are active, which bounds index memory and the second column scan; dense and unsupported layouts keep the existing generic path. Literal dotted keys also keep the generic path so their semantics cannot depend on other rows in the batch. On the fixed 10 GB workload with 2,000 materialized paths and 20 values per row, the wall median decreases from 77.7376 to 75.0729 seconds and BE CPU decreases from 245.04 to 227.51 core-seconds, improvements of 3.43% and 7.15% respectively.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • ASAN BE UT: VariantAssemblerLegacyTest.*, 34/34 passed.
      • Red/green proof: the literal dotted-path regression fails when its eligibility guard is removed and passes with the guard restored.
      • Coverage includes sparse/dense fallback, root and SQL nulls, duplicate/conflicting paths, arrays, legacy encodings, failure atomicity, and literal dotted keys.
    • Manual test
      • One warmup and three measured trials on the fixed 10 GB sparse workload.
    • No need to test or manual test. Explain why:
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: VariantAssembler scanned every flat materialized path for every row even when only a small fraction of paths were present. Build a batch-local active-slot index for the narrow whole-root, flat, non-conflicting layout and keep all other layouts on the existing generic path. On the fixed 10 GB workload with 2000 materialized paths and 20 values per row, the wall median decreases from 77.7376 to 75.0729 seconds and BE CPU decreases from 245.04 to 227.51 core-seconds, improvements of 3.43% and 7.15% respectively.

### Release note

None

### Check List (For Author)

- Test: Unit Test and manual performance validation
    - ASAN BE UT: VariantAssemblerLegacyTest.*, 33/33 passed
    - Performance benchmark: one warmup and three trials on the fixed 10 GB sparse workload
- Behavior changed: No
- Does this need documentation: No
@eldenmoon
eldenmoon force-pushed the branch-variant-v2-direct-assembler branch from c67cf42 to 3f3af78 Compare September 2, 2026 04:11
@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@eldenmoon eldenmoon closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants