CNS-139: docs: document query history and statement logging for self-managed - #38416
Open
jubrad wants to merge 5 commits into
Conversation
Self-managed operators had no guidance on query history, which the Helm chart now enables by default with a capped sample rate. Add a Query History page under self-managed-deployments covering the shipped default, how to tune the cap via the operator chart value and via system parameters, and the CPU/storage cost of raising it. Because the chart value is passed as --system-parameter-default, a ConfigMap or ALTER SYSTEM SET value overrides it, so the page spells out that precedence and how to undo an override. Also cross-link from the console monitoring page, note the parameter in the system-parameters guide, and stop mz_internal.md from implying the max sample rate is Materialize-controlled only. CNS-139
Address accuracy issues found reviewing the query history docs against the source: - Sampling is not the only gate. statement_logging_target_data_rate throttles logging by byte rate, so a sample rate of 1.0 does not record every statement. - ALTER SYSTEM SET on statement_logging_max_sample_rate requires mz_system, not any superuser: the parameter is not user-modifiable. - A chart value change needs a requested rollout, since upgrading the operator does not roll out instances by itself. - The statement history collections are never truncated, so their storage grows for the life of the instance even though mz_recent_activity_log only surfaces 24 hours. - Self-managed logs mz_system statements by default, which affects the cost of raising the rate. Also fix transform-data/troubleshooting.md, which claimed the sample rate is not user-configurable and gave a three-day window, and drop the ConfigMap example duplicated from the query history page. CNS-139
CNS-136 also exposes operator.args.statementLoggingTargetDataRate. It is the byte-rate cap that actually bounds history growth on busy instances, so document it alongside the sample rate rather than mentioning the throttle only in passing. CNS-139
CNS-136 changed the chart sample rate default from 0.1 to 0.99, matching Cloud, and clarified that the sample rate is not what bounds how much query history stores. Update the documented defaults and invert the tuning guidance to match: statement_logging_target_data_rate is the parameter that bounds storage growth, so that is the one to lower on instances with limited storage. Lowering the sample rate only reduces environmentd CPU overhead and makes the history less representative, without lowering the ceiling. CNS-139
The Cloud sample rate and data rate can change without a PR and without notifying customers, so documenting that a self-managed deployment behaves the same as Cloud is a claim we cannot hold to, and one that would go stale silently. State the shipped defaults on their own terms instead. CNS-139
jubrad
marked this pull request as ready for review
August 25, 2026 02:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DNM until 9/2/26
Query history is becoming available and enabled by default in self-managed deployments, but the docs said nothing about it. Self-managed operators had no guidance on the shipped sample rate, how to change it, or what it costs.
Linear: https://linear.app/materializeinc/issue/CNS-139/docs-document-query-history-and-statement-logging-for-self
Changes
self-managed-deployments/query-history.md: what query history is and who can see it, the two parameters that bound how much is collected, how to tune each via the Helm chart and via system parameters, and the CPU/storage cost of each.console/monitoring.md: notes that query history is sampled and links to the new page.reference/system-catalog/mz_internal.md: no longer says the max sample rate is "set by Materialize and cannot be changed by users". Now distinguishes Cloud (Materialize-controlled) from self-managed (operator-controlled).transform-data/troubleshooting.md: this page claimed outright that "the sampling rate is not user-configurable", which directly contradicts the new page. Also corrected its "three days" window to 24 hours, which is whatmz_recent_activity_logactually retains.configuration-system-parameters.md: adds both statement-logging parameters to the parameter list.Notes for the reviewer
Merge after #38406 (CNS-136). That PR exposes the chart values this page documents. Until it lands, the chart still passes
--disable-statement-logging, so "enabled by default" describes the post-#38406 chart, notmaintoday. Documented values track #38406's latest revision:statementLoggingMaxSampleRate: 0.99andstatementLoggingTargetDataRateunset (inheriting 2071 B/s). The console side is #38407/#38408.Two behaviors are worth a careful look, since both are easy to get wrong and I verified them against the source rather than assuming:
environmentdas--system-parameter-default=..., so they are only defaults. A ConfigMap orALTER SYSTEM SETvalue is upserted into the catalog and wins. Undoing an override needs both removing the ConfigMap key andALTER SYSTEM RESET: removing the key alone leaves the last synced value in the catalog, and resetting alone gets reapplied by the 1s sync loop.statement_logging_target_data_ratethrottles sampled statements, so a sample rate of1.0does not record everything, and lowering the sample rate does not lower the ceiling on what statement logging stores. Relatedly, the statement history collections are never truncated, so storage grows for the life of the instance even though the view only shows 24 hours. The page therefore points operators at the data rate for storage concerns and at the sample rate only forenvironmentdCPU.ALTER SYSTEM SETfor these parameters requiresmz_system, not just any superuser, since they are not user-modifiable. The page says so.Testing
ci/test/lint-docs.shlocally: both hugo builds clean,htmltestreports only 2 pre-existing AWS Glue link errors also present on the base commit, andlint-docs-catalog.shshows no.sltdrift from themz_internal.mdedits.🤖 Generated with Claude Code