SQL-204: only trust forwarded client IPs behind a proxy - #38442
Draft
jubrad wants to merge 1 commit into
Draft
Conversation
A client reaching environmentd directly could set the `mz_forwarded_for` pgwire startup parameter, or prepend a PROXY protocol v2 header on HTTP, and have that address recorded as the session's client IP. Network policies are evaluated against that address, so any client could claim to connect from an allowed one. Add a per-listener `behind_trusted_proxy` flag, defaulting to false, and honor `mz_forwarded_for`, `mz_connection_uuid`, and the PROXY protocol header only where it is set. orchestratord sets it on the external listeners, which balancerd fronts, and only when it also creates balancerd. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://linear.app/materializeinc/issue/SQL-204/network-policies-can-be-circumvented-if-you-have-direct-environmentd
Issue
A client reaching
environmentddirectly can set themz_forwarded_forpgwire startup parameter, or prepend a PROXY protocol v2 header on HTTP, and have that address recorded as the session's client IP. Network policies are evaluated against that address, so any client can claim to connect from an allowed one.environmentdtrusted the value on every listener, with no check that the connection came frombalancerd.Solution
Add a per-listener
behind_trusted_proxyflag tomz_server_core::listeners::{BaseListenerConfig, HttpListenerConfig}, threaded intomz_pgwire::ConfigandHttpConfig.mz_forwarded_for,mz_connection_uuid, and the PROXY protocol header are honored only where it is set; elsewhere the direct peer address is used. The startup parameters are removed from the params map either way, so they never reach session var initialization.The flag is
#[serde(default)]false, so both listener config schema versions parse unchanged and every hand-written config is safe by default.orchestratordsets it on theexternalSQL and HTTP listeners, whichbalancerdfronts, and only when it also createsbalancerd.internalandmetricslisteners never get it.Tests
test_forwarded_client_ip_requires_trusted_proxy(src/environmentd/tests/pgwire.rs): a startup packet carryingmz_forwarded_foryields the real peer IP on an untrusted listener and the forwarded IP on a trusted one.test_proxy_header_requires_trusted_proxy(src/environmentd/tests/server.rs): same for a PROXY v2 header over HTTP.src/server-core/src/listeners.rs: a config that omits the field parses as false.test/balancerd:no_auth_https.jsonandfrontegg_https.jsonmark their external listeners as proxy-fronted. The two PROXY-header workflows moved fromenvironmentd's internal HTTP port to the external one, the only portbalancerdactually fronts, which means they now send the header ahead of the TLS handshake asbalancerddoes.For the reviewer
This does not close the hole on the
balancerd-fronted external listener itself, which still trusts anything that reaches it directly. In cloud that path is blocked at the network layer. Tightening it further (a peer-address allowlist, or a sharedbalancerd<->environmentdsecret) is a separate design;balancerdconnects withSslVerifyMode::NONE, so there is no client identity to verify today.A self-managed deployment that fronts
environmentdwith its own proxy instead ofbalancerdwill now see that proxy's address as the client IP.create_balancersis the only signalorchestratordhas for this.test/balancerdis nightly-only, so its changes are not covered by this PR's checks.🤖 Generated with Claude Code