[DNM] test rate-limited Leader return to a restarted TiKV store

[exit-code-1]

What problem does this PR solve?

Warning

DNM / test-only / do not merge. This branch exists only for staging experiments for tidbcloud/cloud-storage-engine#5774.

Issue Number: N/A (DNM test PR)

After a TiKV restart and removal of evict-leader, automatic leader balancing can return many Leaders to the restarted target Store in a short interval. This makes it hard to separate delayed admission from paced admission when measuring cold working-set/runtime I/O, Prewrite latency, business tail latency, and QPS.

What is changed and how does it work?

Add an opt-in leaders-per-second token bucket to balance-leader-scheduler, keyed by target store ID.

Keep the default unlimited, persist and hot-reload the per-target settings, expose exact readback, and add pd-ctl set/delete commands. Fix burst at one Leader so a configured target cannot receive a scheduler batch at once.

Configuration example:

pd-ctl scheduler config balance-leader-scheduler \
  set-inbound-leader-transfer-rate 42 2

pd-ctl scheduler config balance-leader-scheduler show

pd-ctl scheduler config balance-leader-scheduler \
  delete-inbound-leader-transfer-rate 42

Readback for the example:

{
  "inbound-leader-transfer-rate-limits": {
    "42": {
      "leaders-per-second": 2,
      "burst": 1
    }
  }
}

Semantics and scope:

• Unit: successfully constructed automatic balance-leader transfer operators per second, aggregated by target store_id.
• Burst is fixed at 1. A newly configured target starts with one token; updates preserve bucket state; deletion restores unlimited behavior immediately.
• The default is fully disabled: an absent target Store entry follows existing behavior.
• Only balance-leader-scheduler automatic transfers are limited. Manual admin-transfer-leader, scatter/balance-range, hot-region, label/rule-checker, evict/grant/shuffle, and witness transfer paths bypass this test limiter.
• Both PD monolith and scheduling service/MCS use the same scheduler implementation. Scheduler config persists through the existing PD scheduler-config storage, and the MCS watcher reload path is covered by unit tests.
• Normal scheduling consumes target tokens even when diagnostic plan collection is enabled. Only an explicit diagnostic dry run bypasses token consumption.
• Failed/duplicate candidate construction does not consume tokens. A unique operator is charged when returned by the scheduler; later waiting-queue rejection or cancellation is not refunded, and normal token refill continues.
• No TiKV Raft message is dropped and no existing Leader, Region, or peer state is mutated by limiter rejection.

Staging comparison:

1. Delayed return: keep evict-leader until the readiness/I/O gate, then remove it with no per-target rate entry.
2. Delayed bulk return: same gate, then allow the current balance-leader scheduler speed without the limiter.
3. Delayed paced return: set the target Store rate before removing evict-leader, then verify readback and observe Leader count, runtime I/O, Prewrite/tail latency, and QPS.

This PR does not start or submit any staging/TCMS execution.

Check List

Tests

• Unit test
  • default unlimited and per-target isolation
  • dynamic set/update/delete and persisted readback
  • token refill and fixed burst
  • diagnostic-enabled executable scheduling versus explicit dry-run behavior
  • persistence-failure rollback for set/update/delete
  • failed candidate and cancellation behavior
  • scheduler-level same-window target limiting
  • scheduling-service watcher ReloadConfig path
• Manual test
  • make pd-ctl
  • pd-ctl command help/readback shape

Code changes

• Has the configuration change
• Has HTTP APIs changed
• Has persistent data change

Side effects

• Increased code complexity

Validation

• make gotest GOTEST_ARGS=\"./pkg/schedule/schedulers -count=1\"
• make gotest GOTEST_ARGS=\"./pkg/schedule/schedulers -run TestInboundLeaderTransferRateLimit -count=1\"
• make gotest GOTEST_ARGS="./pkg/schedule/schedulers -run TestBalanceLeaderInboundTargetRateLimit -count=1"
• GOTOOLCHAIN=go1.25.12 go test -race ./pkg/schedule/schedulers -run TestBalanceLeaderInboundTargetRateLimit -count=1
• root scheduler-package golangci-lint: 0 issues
• tools-module make static GO_EFFECTIVE_TOOLCHAIN=go1.25.12: 0 issues
• GOTOOLCHAIN=go1.25.12 go test ./pd-ctl/pdctl/command -count=1
• make pd-ctl
• git diff --check

Release note

None.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign zhouqiang-cl for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

Hi @exit-code-1. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a37e55ce-8cff-47d8-9b70-78855415aaaf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-linked-issue label, please provide the linked issue number on one line in the PR body, for example: Issue Number: close #123 or Issue Number: ref #456, multiple issues should use full syntax for each issue and be separated by a comma, like: Issue Number: close #123, ref #456.

_{📖 For more info, you can check the "Linking issues" section in the CONTRIBUTING.md.}