Skip to content

CNS-139: docs: document query history and statement logging for self-managed - #38416

Open
jubrad wants to merge 5 commits into
MaterializeInc:mainfrom
jubrad:justin/cns-139-docs-document-query-history-and-statement-logging-for-self
Open

CNS-139: docs: document query history and statement logging for self-managed#38416
jubrad wants to merge 5 commits into
MaterializeInc:mainfrom
jubrad:justin/cns-139-docs-document-query-history-and-statement-logging-for-self

Conversation

@jubrad

@jubrad jubrad commented Aug 22, 2026

Copy link
Copy Markdown
Member

DNM until 9/2/26

Query history is becoming available and enabled by default in self-managed deployments, but the docs said nothing about it. Self-managed operators had no guidance on the shipped sample rate, how to change it, or what it costs.

Linear: https://linear.app/materializeinc/issue/CNS-139/docs-document-query-history-and-statement-logging-for-self

Changes

  • New page self-managed-deployments/query-history.md: what query history is and who can see it, the two parameters that bound how much is collected, how to tune each via the Helm chart and via system parameters, and the CPU/storage cost of each.
  • console/monitoring.md: notes that query history is sampled and links to the new page.
  • reference/system-catalog/mz_internal.md: no longer says the max sample rate is "set by Materialize and cannot be changed by users". Now distinguishes Cloud (Materialize-controlled) from self-managed (operator-controlled).
  • transform-data/troubleshooting.md: this page claimed outright that "the sampling rate is not user-configurable", which directly contradicts the new page. Also corrected its "three days" window to 24 hours, which is what mz_recent_activity_log actually retains.
  • configuration-system-parameters.md: adds both statement-logging parameters to the parameter list.

Notes for the reviewer

Merge after #38406 (CNS-136). That PR exposes the chart values this page documents. Until it lands, the chart still passes --disable-statement-logging, so "enabled by default" describes the post-#38406 chart, not main today. Documented values track #38406's latest revision: statementLoggingMaxSampleRate: 0.99 and statementLoggingTargetDataRate unset (inheriting 2071 B/s). The console side is #38407/#38408.

Two behaviors are worth a careful look, since both are easy to get wrong and I verified them against the source rather than assuming:

  1. Precedence. The chart values reach environmentd as --system-parameter-default=..., so they are only defaults. A ConfigMap or ALTER SYSTEM SET value is upserted into the catalog and wins. Undoing an override needs both removing the ConfigMap key and ALTER SYSTEM RESET: removing the key alone leaves the last synced value in the catalog, and resetting alone gets reapplied by the 1s sync loop.
  2. The sample rate is not the storage lever. statement_logging_target_data_rate throttles sampled statements, so a sample rate of 1.0 does not record everything, and lowering the sample rate does not lower the ceiling on what statement logging stores. Relatedly, the statement history collections are never truncated, so storage grows for the life of the instance even though the view only shows 24 hours. The page therefore points operators at the data rate for storage concerns and at the sample rate only for environmentd CPU.

ALTER SYSTEM SET for these parameters requires mz_system, not just any superuser, since they are not user-modifiable. The page says so.

Testing

ci/test/lint-docs.sh locally: both hugo builds clean, htmltest reports only 2 pre-existing AWS Glue link errors also present on the base commit, and lint-docs-catalog.sh shows no .slt drift from the mz_internal.md edits.

🤖 Generated with Claude Code

jubrad added 5 commits August 21, 2026 20:48
Self-managed operators had no guidance on query history, which the Helm
chart now enables by default with a capped sample rate.

Add a Query History page under self-managed-deployments covering the
shipped default, how to tune the cap via the operator chart value and via
system parameters, and the CPU/storage cost of raising it. Because the
chart value is passed as --system-parameter-default, a ConfigMap or
ALTER SYSTEM SET value overrides it, so the page spells out that
precedence and how to undo an override.

Also cross-link from the console monitoring page, note the parameter in
the system-parameters guide, and stop mz_internal.md from implying the
max sample rate is Materialize-controlled only.

CNS-139
Address accuracy issues found reviewing the query history docs against
the source:

- Sampling is not the only gate. statement_logging_target_data_rate
  throttles logging by byte rate, so a sample rate of 1.0 does not
  record every statement.
- ALTER SYSTEM SET on statement_logging_max_sample_rate requires
  mz_system, not any superuser: the parameter is not user-modifiable.
- A chart value change needs a requested rollout, since upgrading the
  operator does not roll out instances by itself.
- The statement history collections are never truncated, so their
  storage grows for the life of the instance even though
  mz_recent_activity_log only surfaces 24 hours.
- Self-managed logs mz_system statements by default, which affects the
  cost of raising the rate.

Also fix transform-data/troubleshooting.md, which claimed the sample
rate is not user-configurable and gave a three-day window, and drop the
ConfigMap example duplicated from the query history page.

CNS-139
CNS-136 also exposes operator.args.statementLoggingTargetDataRate. It is
the byte-rate cap that actually bounds history growth on busy instances,
so document it alongside the sample rate rather than mentioning the
throttle only in passing.

CNS-139
CNS-136 changed the chart sample rate default from 0.1 to 0.99, matching
Cloud, and clarified that the sample rate is not what bounds how much
query history stores.

Update the documented defaults and invert the tuning guidance to match:
statement_logging_target_data_rate is the parameter that bounds storage
growth, so that is the one to lower on instances with limited storage.
Lowering the sample rate only reduces environmentd CPU overhead and
makes the history less representative, without lowering the ceiling.

CNS-139
The Cloud sample rate and data rate can change without a PR and without
notifying customers, so documenting that a self-managed deployment
behaves the same as Cloud is a claim we cannot hold to, and one that
would go stale silently.

State the shipped defaults on their own terms instead.

CNS-139
@jubrad
jubrad marked this pull request as ready for review August 25, 2026 02:32
@jubrad
jubrad requested a review from a team as a code owner August 25, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant