Skip to content

Commit 1983bf1

Browse files
fix(client): don't skip live changes after transaction ID wraparound (#4761)
Fixes subset snapshot filtering when PostgreSQL 32-bit transaction IDs wrap across an xid8 epoch. Clients now keep live collections current instead of discarding valid changes after wraparound, including on quiet SSE streams that have not received another up-to-date boundary. ## Root Cause Change messages carry 32-bit transaction IDs, while subset snapshot metadata uses 64-bit epoch-aware IDs. Comparing them directly loses the epoch and can classify a new wrapped transaction as visible in an old snapshot. Reconstructing the epoch also has a half-range constraint: the 32-bit xid and its 64-bit reference must be within `2^31` transactions. A quiet stream could retain a snapshot past that bound because its next change may arrive before another `up-to-date` message. ## Approach - Resolve every wire xid into the epoch nearest the snapshot `xmax`, then use the latest resolved xid for snapshot eviction and duplicate filtering. - Advance snapshot retirement in message order from both `global_last_seen_lsn` control messages and each change message `lsn`. - Expand the model-based tests across xid8 epochs, wrap boundaries, and multi-xid messages. - Add regressions for wrapped xid eviction and for a quiet stream whose first later change passes the snapshot WAL position. This mirrors the xid reconstruction strategy already used by the sync service in [#2320](#2320). ## Key Invariants - An active snapshot never outlives the point where the stream WAL position passes its `database_lsn`. - Changes before that boundary still receive snapshot duplicate filtering; later changes do not. - Each wire xid is resolved before the latest contributing transaction is chosen. ## Non-goals - No sync-service or wire-protocol changes. - No changes to subset snapshot semantics outside transaction ID resolution and retirement. ## Trade-offs Using `xmax` as the epoch reference keeps the fix local and matches PostgreSQL wraparound arithmetic. Its half-range requirement is safe because LSN-based retirement bounds each snapshot lifetime without adding protocol state. ## Verification ```bash pnpm --dir packages/typescript-client test --run pnpm --dir packages/typescript-client typecheck pnpm --dir packages/typescript-client stylecheck ``` The full TypeScript client unit suite passed (440 tests), along with focused regression tests, type checking, linting, and formatting checks. ## Files changed - `packages/typescript-client/src/client.ts` — retires snapshots from control and change LSNs in stream order. - `packages/typescript-client/src/snapshot-tracker.ts` — reconstructs epoch-aware transaction IDs before filtering. - `packages/typescript-client/test/pbt-micro.test.ts` — adds cross-epoch property coverage and stream retirement regression. - `packages/typescript-client/test/snapshot-tracker.test.ts` — adds a wrapped xid retirement regression. - `.changeset/tidy-xids-wrap.md` — records the client patch. --- ## Related Related: [#2320](#2320) _AI disclosure: Codex helped investigate, test, and prepare this change._ --------- Co-authored-by: Kyle Mathews <mathews.kyle@gmail.com>
1 parent 8cabe9b commit 1983bf1

5 files changed

Lines changed: 349 additions & 34 deletions

File tree

.changeset/tidy-xids-wrap.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
"@electric-sql/client": patch
3+
---
4+
5+
Fix subset snapshot filtering after PostgreSQL transaction ID wraparound and
6+
retire filters once the stream passes each snapshot's database LSN.

packages/typescript-client/src/client.ts

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1571,8 +1571,26 @@ export class ShapeStream<T extends Row<unknown> = Row>
15711571
// Filter messages using snapshot tracker
15721572
const messagesToProcess = batch.filter((message) => {
15731573
if (isChangeMessage(message)) {
1574+
const changeLsn = message.headers.lsn
1575+
if (typeof changeLsn === `string` && changeLsn) {
1576+
// A quiet SSE stream may deliver its first later change before the
1577+
// next up-to-date boundary. Retire snapshots against that change's
1578+
// WAL position before resolving its wrapped transaction ID.
1579+
this.#snapshotTracker.lastSeenUpdate(BigInt(changeLsn))
1580+
}
15741581
return !this.#snapshotTracker.shouldRejectMessage(message)
15751582
}
1583+
1584+
if (isUpToDateMessage(message)) {
1585+
const lastSeenLsn = message.headers.global_last_seen_lsn
1586+
if (typeof lastSeenLsn === `string` && lastSeenLsn) {
1587+
// Process this in message order: changes before the up-to-date
1588+
// boundary still need snapshot deduplication, while later changes do
1589+
// not once the database has passed the snapshot's LSN.
1590+
this.#snapshotTracker.lastSeenUpdate(BigInt(lastSeenLsn))
1591+
}
1592+
}
1593+
15761594
return true // Always process control messages
15771595
})
15781596

packages/typescript-client/src/snapshot-tracker.ts

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,31 @@ export class SnapshotTracker {
9191
if (set.size === 0) map.delete(key)
9292
}
9393

94+
/**
95+
* Resolves a 32-bit xid against an epoch-aware xid8.
96+
*
97+
* This signed modulo-2^32 calculation requires the reference and xid to be
98+
* within 2^31 transactions. ShapeStream enforces that lifetime bound by
99+
* retiring snapshots as global_last_seen_lsn passes their database_lsn.
100+
*
101+
* Mirrors `Electric.Postgres.Xid`
102+
* (`packages/sync-service/lib/electric/postgres/xid.ex`).
103+
*/
104+
#toXid8(xid: number, referenceXid8: bigint): bigint {
105+
return referenceXid8 + BigInt.asIntN(32, BigInt(xid) - referenceXid8)
106+
}
107+
108+
/**
109+
* Resolves each contributing xid into the epoch nearest `referenceXid8` and
110+
* returns the latest. Snapshot callers use `xmax` as the reference.
111+
*/
112+
#resolveLatestXid8(xids: number[], referenceXid8: bigint): bigint {
113+
return xids.reduce((latest, xid) => {
114+
const xid8 = this.#toXid8(xid, referenceXid8)
115+
return xid8 > latest ? xid8 : latest
116+
}, BigInt(-1))
117+
}
118+
94119
/**
95120
* Check if a change message should be filtered because its already in an active snapshot
96121
* Returns true if the message should be filtered out (not processed)
@@ -99,19 +124,20 @@ export class SnapshotTracker {
99124
const txids = message.headers.txids || []
100125
if (txids.length === 0) return false
101126

102-
const xid = Math.max(...txids) // Use the maximum transaction ID
103-
104127
for (const [xmax, snapshots] of this.xmaxSnapshots.entries()) {
105-
if (xid >= xmax) {
128+
const xid8 = this.#resolveLatestXid8(txids, xmax)
129+
if (xid8 >= xmax) {
106130
for (const snapshot of snapshots) {
107131
this.removeSnapshot(snapshot)
108132
}
109133
}
110134
}
111135

112-
return [...this.activeSnapshots.values()].some(
113-
(x) => x.keys.has(message.key) && isVisibleInSnapshot(xid, x)
114-
)
136+
return [...this.activeSnapshots.values()].some((snapshot) => {
137+
if (!snapshot.keys.has(message.key)) return false
138+
const xid8 = this.#resolveLatestXid8(txids, snapshot.xmax)
139+
return isVisibleInSnapshot(xid8, snapshot)
140+
})
115141
}
116142

117143
lastSeenUpdate(newDatabaseLsn: bigint): void {

0 commit comments

Comments
 (0)