core, server/cluster: isolate preparing range size scans

[lhy1024]

What problem does this PR solve?

After #11072 removes duplicate scans, each unique non-empty Preparing range still scans the root Region tree. Region heartbeats update the same tree under its write lock, so a large range query can delay routing metadata updates.

Using the existing overlapTree would move contention away from the root tree, but would make range queries contend with all subtree readers and writers under st.

Issue Number: ref #9574, ref #11082

What is changed and how does it work?

Start an eventually consistent `regionSizeTree` with the primary RaftCluster. Its dedicated worker performs the initial full-tree build asynchronously and maintains a compact, independently locked B-tree for approximate range-size queries. Active followers and the independent Scheduling Service do not start or query this index.

After a successful root-tree mutation releases `t`, enqueue the accepted Region ID and any Region IDs removed as overlaps when the range or approximate size changed. A capacity-one notifier wakes the single worker, while a pending-ID map coalesces repeated updates. The worker reloads current root state in batches, so delayed update/delete notifications, split, merge, overlap replacement, and reset converge without applying stale event payloads.

Keep the O(1) full-range Preparing lookup on the root Region tree. Read non-empty ranges from the independent size tree while holding its own read lock for the duration of the query. This lock is independent from both the root and subtree locks, so a range query can delay only the size-tree worker. While the initial build is not ready, defer bounded Preparing threshold and progress evaluation rather than reading a partial index or falling back to a root range scan.

Once ready, Preparing uses the eventually consistent size-tree value directly. Preparing progress and threshold calculations are approximate and do not require a point-in-time root-tree snapshot, so this path does not perform a second root-tree confirmation scan.

The size tree stores compact ID/start/end/size entries rather than retaining full RegionInfo objects. It retains immutable range-key slices captured from root state; a later same-range heartbeat may replace the root RegionInfo without refreshing those slices, so the index can retain one older key pair per Region until a range change, reset, or stop. Repeated same-range heartbeats do not accumulate additional copies.

No size-tree operation waits for its lock while holding the root-tree lock t or subtree lock st. A range query can delay the size-tree worker for the duration of that query, but cannot delay the authoritative root or subtree heartbeat update.

Observability is provided by:

• pd_core_region_size_tree_ready
• pd_core_region_size_tree_pending
• pd_core_region_size_tree_oldest_pending_duration_seconds
• pd_core_region_size_tree_rebuild_duration_seconds

The gauges are sampled by the primary RaftCluster's periodic metrics collector, including when Scheduling Service is independent, rather than updated on the Region-heartbeat path.

Check List

Tests

• Unit test
  • make gotest GOTEST_ARGS='./pkg/core ./server/cluster'
  • make gotest GOTEST_ARGS='./pkg/core ./server/cluster -run "TestRegionSizeTree|TestPreparingRegionSize|TestCheckStoreDefersPreparing" -race -count=3'
  • make gotest GOTEST_ARGS='./pkg/core ./server/cluster -tags=nextgen -run "TestRegionSizeTree|TestPreparingRegionSize|TestCheckStoreDefersPreparing" -count=1'
• Static analysis
  • make check

The tests cover initial build and readiness, batched root access, range-query semantics, context cancellation, reset/rebuild interleavings, bounded pending cancellation, root/subtree lock isolation, pending coalescing, split/merge, delayed notifications, overlap replacement across all root mutation entry points, unavailable Preparing thresholds, and primary-PD metric collection in independent Scheduling Service mode.

Preliminary benchmark

Temporary local benchmark on an AMD Ryzen 7 7840S. The rebuild measurement uses the real batched root ScanRegions path and was taken immediately after rebuild, while captured range-key slices still shared backing arrays with the then-current root RegionInfo.

Regions	Index memory	Initial full rebuild	Bounded scan covering all Regions
100K	107.7 B/Region	51.3 ms	1.63 ms
1M	121.8 B/Region	748 ms	23.8 ms

The 1M memory result is about 116 MiB. A linear 10M extrapolation is about 1.13 GiB. After ordinary same-range heartbeats replace root RegionInfo objects, the index may additionally retain one previous start/end key backing pair per Region. Production-scale heartbeat p99 and churn-inclusive memory benchmarks remain useful for quantifying the total benefit and cost.

Side effects and scope

• Primary lifecycle: every primary RaftCluster builds and maintains one size tree until it stops. Active followers retain only the nil optional pointer and do not run the worker.
• Additional memory: one compact item, one Region-ID map entry, and B-tree storage per Region, plus coalesced pending IDs and at most one retained start/end key backing pair per Region.
• Additional updates: changed Region ranges or approximate sizes enqueue one coalesced ID and cause one asynchronous O(log N) tree reconciliation. Unchanged primary heartbeats perform the optional-index check and size comparison but do not enqueue; followers return after the nil-index check.
• Initial readiness: bounded Preparing threshold and progress evaluation are deferred until the asynchronous initial build completes. The Preparing timeout path is unchanged.
• Consistency: after readiness, range sizes are eventually consistent. Temporary stale-low or stale-high values can advance or delay the approximate Preparing transition until later updates or the existing timeout.
• Query complexity: this PR isolates bounded O(N) scans from root/subtree locks; it does not make arbitrary range aggregation O(log N). Augmented aggregate indexes and reusable multi-statistics support remain follow-up work in core: add an aggregated RegionStats index for arbitrary range queries #11082.

Related change: this builds on the per-round range cache merged in #11072.

Release note

Reduce Region heartbeat delays during store preparation by moving non-empty range-size scans to an independently maintained Region size tree.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

The PR adds an independently locked, generation-aware regionSizeTree for approximate range-size queries. RegionsInfo updates it outside subtree locks. Bounded preparation checks use the size tree, while full-range checks continue to use the root tree.

Region-size index implementation pkg/core/region_size_tree.go, pkg/core/region_size_tree_test.go	The new index supports generation-checked updates and removals, overlap replacement, reset, bounded and full-range totals, scan limits, and concurrency tests.
RegionsInfo integration pkg/core/region.go	RegionsInfo initializes and maintains sizeTree during updates, resets, and removals. It exposes GetRegionSizeByRangeFromSizeTree.
Preparing-size query selection server/cluster/cluster.go, server/cluster/cluster_test.go	checkStores uses root-tree size for unbounded ranges and size-tree size for bounded ranges. Tests verify both paths.
Region-size consistency validation pkg/core/region_test.go	Tests cover eventual consistency, range totals, reference counts, removal, reset behavior, and root-tree equivalence.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant checkStores
  participant RegionsInfo
  participant regionSizeTree
  checkStores->>RegionsInfo: request preparing region size
  alt bounded range
    RegionsInfo->>regionSizeTree: query approximate range size
    regionSizeTree-->>RegionsInfo: return size-tree total
  else unbounded range
    RegionsInfo-->>checkStores: return root-tree total
  end
  RegionsInfo-->>checkStores: return selected region size

Loading

Possibly related issues

• core: add an aggregated RegionStats index for arbitrary range queries #11082 — The PR implements the issue’s independent, eventually consistent range-size index and bounded-range integration.

Possibly related PRs

• tikv/pd#11072 — Both changes modify the checkStores region-size calculation flow in server/cluster/cluster.go.

Suggested reviewers: rleungx

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the affected packages and the main change: isolating Preparing range-size scans.
Description check	✅ Passed	The description covers the problem, implementation, tests, side effects, related changes, and release note required by the template.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 97.98851% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.59%. Comparing base (a187877) to head (7e48f0d).
⚠️ Report is 21 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11098      +/-   ##
==========================================
+ Coverage   79.17%   79.59%   +0.41%     
==========================================
  Files         541      545       +4     
  Lines       76487    78046    +1559     
==========================================
+ Hits        60558    62117    +1559     
+ Misses      11629    11608      -21     
- Partials     4300     4321      +21     

Flag	Coverage Δ
unittests	79.59% <97.98%> (+0.41%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/core/region_size_index.go`:
- Around line 78-90: Prevent stale removals from deleting newer indexed
metadata: in pkg/core/region_size_index.go lines 78-90, update
regionSizeIndex.removeIfCurrent to compare the indexed RegionInfo
version/metadata under i.mu, reject older removal requests, and remove only the
matching indexed state; in pkg/core/region.go lines 1508-1514, preserve the
existing lock-free ordering while pass through the metadata needed so delayed
removal cannot erase an update completed after removeRegionFromSubTree.

In `@pkg/core/region.go`:
- Around line 1203-1206: Update updateSubTreeCacheOrderInsensitive to return
whether the region metadata was accepted, then have the surrounding update flow
call regionSizeIndex.updateIfCurrent only when that result indicates success.
Skip the index write when stale metadata is rejected, while preserving the
existing generation handling and lock-order behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ae712d70-fb1c-4502-b4e7-555291de4805

📥 Commits

Reviewing files that changed from the base of the PR and between 3b53c40 and cd99024.

📒 Files selected for processing (6)

• pkg/core/region.go
• pkg/core/region_size_index.go
• pkg/core/region_size_index_test.go
• pkg/core/region_test.go
• server/cluster/cluster.go
• server/cluster/cluster_test.go

🚧 Files skipped from review as they are similar to previous changes (3)

• server/cluster/cluster.go
• server/cluster/cluster_test.go
• pkg/core/region_test.go

[bufferflies]

Follow-up to my earlier review comment, anchoring the specific suggestions to code (see inline comments below).

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-08-19 03:03:30.43371566 +0000 UTC m=+38845.604809779: ☑️ agreed by bufferflies.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bufferflies]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[JmPotato]

Focused follow-up review of the Region size index update and freshness path. The root-mutation hook and ID-based reconciliation look sound; the inline comments cover the remaining actionable correctness, coverage, and validation issues.

Non-line items: please rewrap the two over-80-character body lines in commit c97a13c2f, and rerun or clearly attribute the currently failing required checks before merge.

[JmPotato]

[P2] Do not promote from an index that may still be behind

isReady() only means that the initial build completed; it does not account for pending or activePending. Under backlog, a stale-low range size can make this comparison pass and ReadyToServeLocked commits the one-way Preparing → Serving transition. Later reconciliation fixes the index but cannot undo that transition. Please require a caught-up/fresh generation for this decision, or perform one authoritative root-tree confirmation when the approximate result would promote the store. Add a pending-backlog test that proves the store is not promoted early.

[lhy1024]

We are intentionally accepting this behavior as part of the contract: Region-size statistics are eventually consistent and Preparing -> Serving uses an approximate threshold. A stale-low value may advance the one-way transition and a stale-high value may delay it; the existing preparing timeout remains the fallback. We will not add a freshness gate or an authoritative root confirmation here, because that would reintroduce the O(N) root-tree scan and the lock contention this PR is intended to isolate. Backlog/oldest-pending metrics are available for operational visibility.

[JmPotato]

[P1] Validate the CPU objective at the reported scale

This remains an O(N) scan; the change isolates it from the Root/Subtree locks but does not by itself prove that the high Preparing CPU reported in #9574 is resolved. The current single-scan microbenchmark does not cover 50 Preparing stores with concurrent Region heartbeats. Please provide a comparative workload measuring total CPU, heartbeat throughput/p99, Root lock wait, memory, and pending convergence lag at million-Region scale before treating the original performance goal as closed.

[lhy1024]

Agreed that a production-scale comparative benchmark would be useful, but this is validation for the broader performance goal rather than a correctness issue in this PR. #11098 specifically decouples bounded range-size reads from the Root/Subtree heartbeat locks; the index query remains O(N), so this PR does not claim to close the total CPU problem from #9574. Million-Region concurrent heartbeat/p99 and pending-lag measurements, together with the O(log N) aggregate design, remain follow-up work tracked in #11082.

[lhy1024]

/test pull-unit-test-next-gen-3

[lhy1024]

/test pull-unit-test-next-gen-2

[lhy1024]

/test pull-integration-realcluster-test

[JmPotato]

Why use an atomic.Pointer while also having a mutex lock?

[lhy1024]

They protect different concerns. sizeTreeMu serializes the compound Start/Stop lifecycle so concurrent lifecycle calls cannot create overlapping workers or stop a worker while it is being started. The atomic.Pointer lets heartbeat mutation, query, reset, and metrics paths observe the optional tree without taking the lifecycle mutex. Using only the mutex would add locking to the heartbeat hot path, while using only the atomic pointer would require a more complex lifecycle state machine.