[improvement](be) Optimize flat Variant assembly for sparse rows - #67423
Closed
eldenmoon wants to merge 1 commit into
Closed
[improvement](be) Optimize flat Variant assembly for sparse rows#67423eldenmoon wants to merge 1 commit into
eldenmoon wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: VariantAssembler scanned every flat materialized path for every row even when only a small fraction of paths were present. Build a batch-local active-slot index for the narrow whole-root, flat, non-conflicting layout and keep all other layouts on the existing generic path. On the fixed 10 GB workload with 2000 materialized paths and 20 values per row, the wall median decreases from 77.7376 to 75.0729 seconds and BE CPU decreases from 245.04 to 227.51 core-seconds, improvements of 3.43% and 7.15% respectively.
### Release note
None
### Check List (For Author)
- Test: Unit Test and manual performance validation
- ASAN BE UT: VariantAssemblerLegacyTest.*, 33/33 passed
- Performance benchmark: one warmup and three trials on the fixed 10 GB sparse workload
- Behavior changed: No
- Does this need documentation: No
eldenmoon
force-pushed
the
branch-variant-v2-direct-assembler
branch
from
September 2, 2026 04:11
c67cf42 to
3f3af78
Compare
Member
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
VariantAssemblerscanned every flat materialized path for every row even when only a small fraction of paths were present. This change builds a batch-local active-slot CSR index only for the narrow whole-root, one-part, non-array, non-conflicting layout with empty root payloads. The fast path is used only when at most 1/16 of row/path cells are active, which bounds index memory and the second column scan; dense and unsupported layouts keep the existing generic path. Literal dotted keys also keep the generic path so their semantics cannot depend on other rows in the batch. On the fixed 10 GB workload with 2,000 materialized paths and 20 values per row, the wall median decreases from 77.7376 to 75.0729 seconds and BE CPU decreases from 245.04 to 227.51 core-seconds, improvements of 3.43% and 7.15% respectively.Release note
None
Check List (For Author)
Test
VariantAssemblerLegacyTest.*, 34/34 passed.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)