Skip to content

Commit b2cf71d

Browse files
alcoclaude
andauthored
Add ELECTRIC_TCP_READ_TIMEOUT for the HTTP keep-alive idle timeout (#4716)
## What Adds `ELECTRIC_TCP_READ_TIMEOUT` to configure ThousandIsland's `read_timeout` (default 60s), which doubles as Bandit's HTTP keep-alive idle timeout: connections that have waited this long for their next request are closed. The option mirrors `ELECTRIC_TCP_SEND_TIMEOUT` (human-readable duration) and threads through `tcp_read_timeout` → `thousand_island_options/1`. Default is unchanged (`nil` → ThousandIsland's 60s). Includes a docs entry in `website/docs/sync/api/config.md` and a changeset. ## Why When Electric runs behind a connection-pooling proxy, the target's keep-alive timeout must **exceed** the proxy's idle timeout. If it doesn't, the proxy can reuse a pooled connection at the same moment Bandit's idle timer closes it (the close is a bare FIN — nothing announces it in-band), and the resulting TCP reset surfaces to clients as an intermittent `502 Bad Gateway`. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent a1026bd commit b2cf71d

6 files changed

Lines changed: 46 additions & 1 deletion

File tree

.changeset/tidy-donuts-brush.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
"@core/sync-service": patch
3+
---
4+
5+
Add `ELECTRIC_TCP_READ_TIMEOUT` to configure the socket read / HTTP keep-alive
6+
idle timeout (ThousandIsland's `read_timeout`, default 60s). When Electric runs
7+
behind a connection-pooling proxy such as an AWS ALB, this must be set above
8+
the proxy's idle timeout — otherwise the proxy races Electric's unannounced
9+
idle close when reusing a pooled connection and clients see intermittent 502s.

packages/sync-service/config/runtime.exs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,8 @@ config :electric,
299299
handler_fullsweep_after: env!("ELECTRIC_TWEAKS_HANDLER_FULLSWEEP_AFTER", :integer, nil),
300300
tcp_send_timeout:
301301
env!("ELECTRIC_TCP_SEND_TIMEOUT", &Electric.Config.parse_human_readable_time!/1, nil),
302+
tcp_read_timeout:
303+
env!("ELECTRIC_TCP_READ_TIMEOUT", &Electric.Config.parse_human_readable_time!/1, nil),
302304
feature_flags: env!("ELECTRIC_FEATURE_FLAGS", &Electric.Config.parse_feature_flags/1, nil),
303305
manual_table_publishing?: env!("ELECTRIC_MANUAL_TABLE_PUBLISHING", :boolean, nil),
304306
publication_refresh_period:

packages/sync-service/lib/electric/application.ex

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,12 @@ defmodule Electric.Application do
366366
send_timeout -> [send_timeout: send_timeout]
367367
end
368368

369+
read_timeout_opts =
370+
case get_env(opts, :tcp_read_timeout) do
371+
nil -> []
372+
read_timeout -> [read_timeout: read_timeout]
373+
end
374+
369375
ipv6_opts =
370376
if get_env(opts, :listen_on_ipv6?) do
371377
[:inet6]
@@ -383,7 +389,7 @@ defmodule Electric.Application do
383389
n -> [genserver_options: [spawn_opt: [fullsweep_after: n]]]
384390
end
385391

386-
acceptor_opts ++ transport_opts ++ genserver_opts
392+
acceptor_opts ++ read_timeout_opts ++ transport_opts ++ genserver_opts
387393
end
388394

389395
defp cowboy_options(opts) do

packages/sync-service/lib/electric/config.ex

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,13 @@ defmodule Electric.Config do
5959
long_poll_timeout: 20_000,
6060
http_api_num_acceptors: nil,
6161
tcp_send_timeout: :timer.seconds(30),
62+
# Socket read/keep-alive timeout (ThousandIsland's read_timeout). Bandit reaps
63+
# keep-alive connections that have been idle for this long. When Electric runs
64+
# behind a connection-pooling proxy (ALB, nginx, etc.), this MUST exceed the
65+
# proxy's idle timeout — otherwise the proxy races Bandit's unannounced close
66+
# when reusing a pooled connection and surfaces the reset as a 502.
67+
# nil keeps ThousandIsland's default (60s).
68+
tcp_read_timeout: nil,
6269
cache_max_age: 60,
6370
cache_stale_age: 60 * 5,
6471
chunk_bytes_threshold: Electric.ShapeCache.LogChunker.default_chunk_size_threshold(),

packages/sync-service/test/electric/config_test.exs

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,15 @@ defmodule Electric.ConfigTest do
5151
Electric.Application.api_server()
5252
end
5353

54+
test "api_server/1 maps tcp_read_timeout to a top-level ThousandIsland option" do
55+
[{Bandit, bandit_opts}] = Electric.Application.api_server(tcp_read_timeout: 180_000)
56+
57+
assert bandit_opts[:thousand_island_options][:read_timeout] == 180_000
58+
59+
[{Bandit, default_opts}] = Electric.Application.api_server([])
60+
refute Keyword.has_key?(default_opts[:thousand_island_options], :read_timeout)
61+
end
62+
5463
test "configuration/1", ctx do
5564
Electric.Application.configuration(
5665
Keyword.take(ctx.initial_config, [:replication_connection_opts])

website/docs/sync/api/config.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,18 @@ This environment variable increases this timeout.
285285

286286
</EnvVarConfig>
287287

288+
### ELECTRIC_TCP_READ_TIMEOUT
289+
290+
<EnvVarConfig
291+
name="ELECTRIC_TCP_READ_TIMEOUT"
292+
defaultValue="60s"
293+
example="180s">
294+
Socket read timeout, which doubles as the HTTP keep-alive idle timeout: Electric closes a connection that has been waiting this long for its next request. Defaults to 60 seconds.
295+
296+
When Electric runs behind a connection-pooling reverse proxy or load balancer (AWS ALB, nginx, etc.), set this **above** the proxy's idle timeout. If the proxy's timeout is longer, the proxy can reuse a pooled connection at the same moment Electric closes it, and the resulting TCP reset surfaces to clients as an intermittent `502 Bad Gateway`. For example, behind an AWS ALB with the default 60-second idle timeout, set `ELECTRIC_TCP_READ_TIMEOUT=120s`.
297+
298+
</EnvVarConfig>
299+
288300
### ELECTRIC_SHAPE_CHUNK_BYTES_THRESHOLD
289301

290302
<EnvVarConfig

0 commit comments

Comments
 (0)