auth: prototype mutual TLS client authentication

Adds mutual TLS as an admission gate for external pgwire connections, enforced
before any credential exchange. Off by default; no behaviour change for
deployments that do not opt in.

Proof of possession and trust evaluation are separated, which is what makes this
work through balancerd. balancerd requests client certificates with a permissive
verify callback, so OpenSSL still validates CertificateVerify (possession is
proven) while the trust decision is deferred to the party that holds the tenant's
anchors. The chain is forwarded to environmentd in a new mz_client_cert startup
parameter as base64 concatenated PEM, leaf first. balancerd never learns any
tenant's certificate authority and stays stateless.

environmentd honours a forwarded chain only from a peer whose own certificate
chains to --tls-proxy-ca, so the balancerd to environmentd leg gains a client
identity (--internal-tls-cert/--internal-tls-key). Without a proxy authority
configured, forwarded certificates are ignored entirely. A client that supplies
mz_client_cert itself is rejected, as with mz_forwarded_for.

Configuration is three system parameters, following the OIDC precedent:
mtls_client_ca (PEM bundle of trust anchors), mtls_mode (disable, allow,
require), and mtls_identity_binding (none, common-name). Anchors and mode are
read from the live ConfigSet rather than via get_system_vars, so the check costs
no coordinator round trip on the connection path.

Two details worth noting. Trust anchors are built with
X509_V_FLAG_PARTIAL_CHAIN, so pinning an intermediate as the sole anchor works
rather than failing with "unable to get local issuer certificate". A leaf with no
Common Name fails a common-name binding outright, because pgwire defaults an
absent user parameter to the empty string and a SPIFFE-shaped certificate would
otherwise satisfy the binding.

Also fixes a pre-existing balancerd bug this work surfaced: the startup parameter
rejection paths called FramedConn::send without flushing, so a client rejected
for supplying mz_connection_uuid or mz_forwarded_for saw a bare connection close
instead of the error.

Adds unit tests for the policy in mz-authenticator, integration tests for the
direct path in src/environmentd/tests/mtls.rs, and tests for the forwarded path
in src/balancerd/tests/server.rs covering anchor rotation, an unauthenticated
proxy being disbelieved, and a forged mz_client_cert being refused.

The HTTP, WebSocket, and webhook paths are in scope for the feature but not yet
wired.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: jubrad

Cargo.lock

@@ -768,6 +768,9 @@ def get_default_system_parameters(
     "oidc_group_role_sync_enabled",
     "oidc_group_claim",
     "oidc_group_role_sync_strict",
+    "mtls_client_ca",
+    "mtls_mode",
+    "mtls_identity_binding",
     "console_oidc_client_id",
     "console_oidc_scopes",
     "enable_public_metrics_endpoint",

misc/python/materialize/mzcompose/__init__.py

@@ -3326,6 +3326,11 @@ def __init__(
             "oidc_group_role_sync_enabled",
             "oidc_group_claim",
             "oidc_group_role_sync_strict",
+            # Flipping an admission policy would sever parallel-workload's own
+            # connections, which present no client certificate.
+            "mtls_client_ca",
+            "mtls_mode",
+            "mtls_identity_binding",
             "console_oidc_client_id",
             "console_oidc_scopes",
             "cluster_controller_tick_interval",

misc/python/materialize/parallel_workload/action.py

@@ -187,6 +187,38 @@ pub const OIDC_GROUP_ROLE_SYNC_STRICT: Config<bool> = Config::new(
     "When true, reject login if OIDC group-to-role sync fails (fail-closed).",
 );
 
+/// Trust anchors for mutual TLS client authentication: a PEM bundle of
+/// certificate authorities whose leaves are accepted.
+///
+/// A bundle rather than a single certificate so that an operator can trust
+/// several authorities at once, and can stage a CA rotation by trusting the old
+/// and the new authority simultaneously.
+pub const MTLS_CLIENT_CA: Config<Option<&'static str>> = Config::new(
+    "mtls_client_ca",
+    None,
+    "PEM bundle of certificate authorities trusted to issue client certificates for mutual TLS.",
+);
+
+/// How strictly to enforce mutual TLS on external logins: `disable`, `allow`,
+/// or `require`. See `mz_authenticator::client_cert::MtlsMode`.
+pub const MTLS_MODE: Config<&'static str> = Config::new(
+    "mtls_mode",
+    "disable",
+    "How strictly to enforce mutual TLS client authentication for external logins: \
+     'disable' ignores client certificates, 'allow' accepts a trusted certificate but does \
+     not require one, 'require' rejects logins without one.",
+);
+
+/// Which certificate field, if any, must agree with the connecting username.
+/// See `mz_authenticator::client_cert::IdentityBinding`.
+pub const MTLS_IDENTITY_BINDING: Config<&'static str> = Config::new(
+    "mtls_identity_binding",
+    "none",
+    "Which client certificate field must match the connecting username: 'none' to treat the \
+     certificate purely as an admission gate, or 'common-name' to require the leaf's Subject \
+     Common Name to equal the username.",
+);
+
 pub const PERSIST_FAST_PATH_ORDER: Config<bool> = Config::new(
     "persist_fast_path_order",
     false,
@@ -452,6 +484,9 @@ pub fn all_dyncfgs(configs: ConfigSet) -> ConfigSet {
         .add(&OIDC_GROUP_ROLE_SYNC_ENABLED)
         .add(&OIDC_GROUP_CLAIM)
         .add(&OIDC_GROUP_ROLE_SYNC_STRICT)
+        .add(&MTLS_CLIENT_CA)
+        .add(&MTLS_MODE)
+        .add(&MTLS_IDENTITY_BINDING)
         .add(&PERSIST_FAST_PATH_ORDER)
         .add(&ENABLE_S3_TABLES_REGION_CHECK)
         .add(&ENABLE_MCP_AGENT)

src/adapter-types/src/dyncfgs.rs

@@ -9,12 +9,15 @@ publish = false
 
 [dependencies]
 jsonwebtoken.workspace = true
+anyhow.workspace = true
 mz-adapter = { path = "../adapter", default-features = false }
 mz-adapter-types = { path = "../adapter-types", default-features = false }
 mz-auth = { path = "../auth", default-features = false }
+mz-dyncfg = { path = "../dyncfg" }
 mz-frontegg-auth = { path = "../frontegg-auth", default-features = false }
 mz-ore = { path = "../ore", features = ["assert"] }
 mz-pgwire-common = { path = "../pgwire-common", default-features = false }
+openssl.workspace = true
 reqwest.workspace = true
 tokio-postgres.workspace = true
 serde.workspace = true