request_throttling.mdx

id	request-throttling
title	Request throttling
description	How to throttle requests per domain using the ThrottlingRequestManager.

import ApiLink from '@site/src/components/ApiLink'; import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

import ThrottlingExample from '!!raw-loader!roa-loader!./code_examples/request_throttling/throttling_example.py';

When crawling websites that enforce rate limits (HTTP 429) or specify crawl-delay in their robots.txt, you need a way to throttle requests per domain without blocking unrelated domains. The ThrottlingRequestManager provides exactly this.

Overview

The ThrottlingRequestManager wraps a RequestManager (typically a RequestQueue) and manages per-domain throttling. You specify which domains to throttle at initialization, and the manager automatically:

• Routes requests for listed domains into dedicated sub-managers at insertion time.
• Enforces delays from HTTP 429 responses (exponential backoff) and robots.txt crawl-delay directives.
• Schedules fairly by fetching from the domain that has been waiting the longest.
• Releases the concurrency slot when all configured domains are throttled, instead of holding it for the whole cooldown.

Requests for domains not in the configured list pass through to the main queue without any throttling.

Basic usage

To use request throttling, create a ThrottlingRequestManager with the domains you want to throttle and pass it as the request_manager to your crawler:

{ThrottlingExample}

How it works

1. Insertion-time routing: When you add requests via add_request or add_requests, each request is checked against the configured domain list. Matching requests go directly into a per-domain sub-manager; all others go to the inner manager. This eliminates request duplication entirely.

2. 429 backoff: When the crawler detects an HTTP 429 response, the ThrottlingRequestManager records an exponential backoff delay for that domain (starting at 2s, doubling up to 60s). If the response includes a Retry-After header, that value takes priority.

3. Crawl-delay: If robots.txt specifies a crawl-delay, the manager enforces a minimum interval between requests to that domain.

4. Fair scheduling: fetch_next_request sorts available sub-managers by how long each domain has been waiting, ensuring no domain is starved.

5. Cooldown handling: While a domain is in a cooldown, its queued requests don't count as dispatchable, so the crawler's autoscaled pool idles instead of keeping a worker slot blocked. The requests still count towards completion, so the crawl waits for them and finishes only once every one has been handled.

:::tip

The ThrottlingRequestManager is an opt-in feature. If you don't pass it to your crawler, requests are processed normally without any per-domain throttling.

:::