fix(throttling-manager): Stop holding a concurrency slot while a domain is throttled

[Mantisus]

Description

• fetch_next_request now returns None instead of sleeping out a cooldown, and is_empty() counts only domains that are not throttled, so the autoscaled pool idles instead of holding worker slots. is_finished() still counts every domain, so the crawl does not end while throttled requests are queued. The old sleep occupied a slot while the system read as idle, which pushed desired_concurrency up during the cooldown and burst the moment it expired.

Issues

• Closes: ThrottlingRequestManager.fetch_next_request holds a concurrency slot while waiting out a throttle #2127

Testing

• Update unit tests

[Copilot]

Pull request overview

Updates throttling behavior so cooldowns no longer occupy crawler concurrency slots.

Changes:

• Return immediately when only throttled requests remain.
• Treat throttled queues as temporarily empty while keeping the crawl unfinished.
• Add tests for cooldown, crawl-delay, expiry, and scheduling order.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/crawlee/request_loaders/_throttling_request_manager.py	Skips throttled domains without waiting and updates readiness semantics.
tests/unit/test_throttling_request_manager.py	Verifies non-blocking throttling and domain scheduling behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[vdusek]

LGTM