Skip to content

Commit bedaa57

Browse files
docs: draft MySQL snapshot parallelism page
User-facing draft for the parallel snapshot feature: how key ranges divide across workers, single-column string PK requirement and fallback, upstream connection and statistics considerations, and where to observe progress. Marked private preview.
1 parent a3d533a commit bedaa57

1 file changed

Lines changed: 78 additions & 0 deletions

File tree

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
title: "Snapshot parallelism"
3+
description: "How Materialize parallelizes the initial snapshot of MySQL tables, and how to get the most out of it."
4+
menu:
5+
main:
6+
parent: "mysql"
7+
name: "Snapshot parallelism"
8+
identifier: "mysql-snapshot-parallelism"
9+
weight: 70
10+
---
11+
12+
{{< private-preview />}}
13+
14+
When you create a [MySQL source](/sql/create-source/mysql-v2/), Materialize
15+
performs an initial, snapshot-based sync of the selected tables before it
16+
starts ingesting change events from the binlog. For large tables, this
17+
snapshot dominates the time until the source becomes healthy and your queries
18+
return up-to-date results.
19+
20+
To speed this up, Materialize can split the snapshot of a single table across
21+
all the workers of the cluster hosting the source. Each worker reads a
22+
disjoint range of the table's primary key space in parallel, instead of one
23+
worker reading the whole table on its own.
24+
25+
## How it works
26+
27+
Before reading a table, Materialize samples the table's primary key to find
28+
boundary keys that divide it into ranges of roughly equal size, using
29+
inexpensive index probes and optimizer row estimates. Each worker then reads
30+
only its assigned key range, within the same consistent snapshot of the
31+
upstream database. The results are identical to a single-worker snapshot,
32+
including transactional consistency: parallelism changes only how fast the
33+
snapshot completes.
34+
35+
Sampling adds a small number of point queries per table before the snapshot
36+
starts. These queries use the primary key index and do not scan the table.
37+
Their number is capped in proportion to the table's estimated size, so
38+
sampling stays negligible next to the snapshot itself.
39+
40+
## Requirements
41+
42+
A table's snapshot is parallelized when all of the following hold:
43+
44+
- The table has a **single-column primary key** of a **string type**
45+
(`CHAR`, `VARCHAR`, or `TEXT`). Composite and numeric primary keys are not
46+
yet supported.
47+
- The table is large enough to be worth splitting. Small tables are read by a
48+
single worker, where parallelism would add overhead without benefit.
49+
50+
Tables that don't meet these requirements are still snapshot correctly, each
51+
by a single worker. Different tables are always processed concurrently,
52+
independent of this feature.
53+
54+
## Upstream considerations
55+
56+
- **Connection count.** During snapshotting, Materialize opens one connection
57+
per worker reading a key range, plus a small number of coordination
58+
connections. If your MySQL server or connection pooler enforces a low
59+
[`max_connections`](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_connections)
60+
limit, account for this burst when sizing it. After the snapshot completes,
61+
the source drops back to a single replication connection.
62+
63+
- **Statistics freshness.** Range boundaries are placed using the MySQL
64+
optimizer's row estimates. Stale statistics don't affect correctness, but
65+
can skew how evenly work divides across workers. Running
66+
[`ANALYZE TABLE`](https://dev.mysql.com/doc/refman/8.0/en/analyze-table.html)
67+
on very large tables before creating the source can improve balance.
68+
69+
- **Read load.** A parallel snapshot reads the same total data as a serial
70+
one, but over a shorter window, so expect proportionally higher read
71+
throughput on the upstream database (or read replica) while it runs.
72+
73+
## Observability
74+
75+
The progress of an ongoing snapshot is visible in the
76+
[`mz_internal.mz_source_statistics`](/reference/system-catalog/mz_internal/#mz_source_statistics)
77+
system catalog view, including the number of rows read so far relative to the
78+
estimated total.

0 commit comments

Comments
 (0)