fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After

[Mantisus]

Description

• Only the first 429 of a burst now advances the exponent, and it decays once the domain has left us alone for a full extra backoff window. Previously every concurrent 429 stepped it, so eight in-flight requests pushed the delay from 2s to past max_delay in one go, and any handled request reset the counter, including one that had just failed for good. throttled_until split into separate backoff_until and crawl_delay_until clocks, since the crawl delay is armed on every dispatch and would otherwise pass for an active backoff.

Issues

• Closes: ThrottlingRequestManager computes the wrong backoff for concurrent 429s and for Retry-After: 0 #2125

Testing

• Added new unit tests

[vdusek]

@janbuchar Does #2158 (comment) apply to this PR as well, or can this one be reviewed and merged?

[janbuchar]

@vdusek I believe this one is free to go. I'll look into it.

[Copilot]

Pull request overview

Fixes throttling backoff behavior for concurrent 429 responses and zero-valued Retry-After headers.

Changes:

• Separates 429 backoff and crawl-delay clocks.
• Adds burst-aware backoff counting and decay.
• Expands unit coverage for throttling edge cases.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/crawlee/request_loaders/_throttling_request_manager.py	Revises backoff state and scheduling.
src/crawlee/_utils/http.py	Treats non-positive retry delays as unusable.
tests/unit/test_throttling_request_manager.py	Tests concurrent 429s, decay, and zero retry delays.

Suppressed comments (2)

src/crawlee/request_loaders/_throttling_request_manager.py:314

• This early return also discards a positive Retry-After from later responses in the same burst. For example, if the first 429 starts the 2-second default backoff and a concurrent response then specifies 30 seconds, the domain is retried after 2 seconds despite the server's header. Keep the exponent unchanged for the burst, but still extend the active deadline to the later of the current deadline and the capped Retry-After deadline (including the existing cap warning).

        if now < state.backoff_until:
            return True

src/crawlee/request_loaders/_throttling_request_manager.py:327

• The exponential multiplication can overflow before the subsequent max_delay cap is applied. With the default delays, a continuously rate-limited domain reaches this after roughly 47 backoff events, at which point timedelta * int raises OverflowError and can abort crawling. Saturate overflow before applying the existing cap.

            delay = self._base_delay * (2 ** (state.consecutive_429_count - 1))

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.