Domain config requestTimeoutMs, priority and isEnabled are accepted, persisted and documented but never applied #54

What happened?

Three fields of DomainConfig are accepted by the API, validated, defaulted, persisted to Postgres, returned on reads, and documented in docs/domain-configs.md — but have no effect on scraping. They share one root cause: internal/processor/processor.go reads the config but doesn't apply these parts of it.

1. requestTimeoutMs is written to a field nobody reads

processor.go:105-106 transfers it onto the handler request:

if domainCfg.RequestTimeoutMs > 0 {
    req.Timeout = time.Duration(domainCfg.RequestTimeoutMs) * time.Millisecond
}

models.HandlerRequest.Timeout (internal/models/types.go:127) is then never read by anything. Grepping the handler package:

• handler/http.go:83 uses h.timeout, the value captured in NewHTTPHandler at construction
• handler/browser.go:86 uses h.timeout likewise
• handler/api.go:75-84 uses its own cfg.Timeout

So every domain gets the process-wide timeout regardless of configuration. docs/domain-configs.md:56 documents the field as "Timeout per handler attempt (ms)", and internal/http/handlers/domain_config.go:67-69 defaults it to 30000 on create, so users have every reason to believe it works.

Related and worth fixing together: NewHTTPHandler is constructed with cfg.BrowserTimeout at cmd/server/main.go:78, so the HTTP handler's timeout is controlled by the BROWSER_TIMEOUT env var. There is no separate knob for the non-browser path.

2. priority is loaded, then discarded

internal/domain/repository.go:29 sorts by it:

FROM domain_configs ORDER BY priority DESC, domain

Cache.refresh (internal/domain/cache.go:63-71) then drops the ordered slice into map[string]*DomainConfig keyed by domain, where ordering is meaningless. GetConfig (cache.go:76-100) does an exact-match lookup, then walks parent domains and returns the first match by proximity, never consulting Priority.

docs/domain-configs.md:179 promises "if multiple configs could match, higher priority wins" and lists it in the field table. As written, priority is inert; the nearest ancestor always wins.

3. isEnabled is bypassed by two of the three things it should gate

processor.go:101 correctly gates handler chain, timeouts, headers, user-agent and proxy:

if domainCfg != nil && domainCfg.IsEnabled {

But two earlier/later checks omit it:

• Blocking, processor.go:88-90 — if domainCfg != nil && domainCfg.Blocked. A config with isEnabled: false still blocks every request to that domain.
• Content validation, processor.go:146 — if domainCfg != nil && p.detector != nil. A disabled config still enforces failurePatterns, requiredPatterns and minContentLength, and still forces retries when they don't match.

docs/domain-configs.md:52 describes the field as "Whether this config is active." Setting it to false is the natural way to temporarily park a config, and today it silently leaves the two most disruptive behaviours switched on — with no way to tell from the API response that this is happening.

What did you expect?

• requestTimeoutMs should bound the individual handler attempt — most directly by having handlers honour req.Timeout when non-zero (falling back to their constructed default), or by deriving a per-attempt context.WithTimeout in Chain.Execute.
• priority should break ties when several configs could match a host, or be dropped from the schema and docs if the exact-then-ancestor walk is considered sufficient.
• isEnabled: false should make the config completely inert — no blocking, no content validation, nothing.

Suggested split

Happy to send this as one PR, but it splits cleanly into three if you'd prefer smaller reviews — (3) is the smallest and highest-severity, (1) touches the handler interface most, (2) is partly a product decision about whether priority is worth keeping at all. Let me know which shape you want and I'll open PRs accordingly. internal/domain is currently at 18.4% coverage (only detector.go is tested — cache.go and repository.go have none), so I'd add matching tests either way.

Steps to reproduce

For (3), which needs no proxies:

1. make up
2. Create a disabled but blocking config:

   curl -X POST localhost:8080/v1/domain-configs -H "Content-Type: application/json" \
     -d '{"domain":"example.com","isEnabled":false,"blocked":true,"blockedReason":"parked"}'

3. Wait up to 60s for the cache refresh (cache.go:34)
4. curl -X POST localhost:8080/v1/scrape -H "Content-Type: application/json" -d '{"url":"https://example.com"}'
5. The job fails with domain is blocked: parked, despite the config being disabled

For (1): set requestTimeoutMs: 1 on a domain and observe that scrapes still take as long as BROWSER_TIMEOUT allows rather than failing after 1ms.

Environment

• OS: macOS 26.5.2 (arm64)
• Go: go1.26.5 darwin/arm64
• Docker version: 29.4.1
• AnakinScraper version: commit c875d37