Delivery guarantees

Data Sync targets exactly-once delivery. That phrase is used loosely in this industry, so this page states precisely what it means here, what it is built from, and where it stops being true. If you are designing something that depends on the guarantee, read the whole page — particularly the last section.

Idempotent writes Checkpoint after flush FIFO per sink

What exactly-once means here

There is no distributed transaction spanning the source read and the sink write. Data Sync does not two-phase commit across your Postgres cluster and your warehouse, and no system that reads a replication log and writes to a separate database can. Anyone who tells you otherwise is describing something else.

The guarantee is assembled from two properties instead:

The checkpoint advances only after a sink flush durably succeeds. A batch is not considered delivered until the sink has acknowledged it durably; only then does the position move.
Sinks write idempotently. Applying the same change twice produces the same row state as applying it once.

Put those together and the failure behaviour follows. A crash — a runner dying, a network partition, a sink timing out — leaves the checkpoint somewhere at or before the last durably written change. On restart, Data Sync resumes from the checkpoint and replays a bounded window of changes that were already applied. The sink absorbs them. The end state is identical to a run that never crashed.

So the accurate description is: at-least-once delivery into an idempotent sink, which converges on exactly-once end state. That is a real and useful guarantee. It is not the same as each row being physically transmitted exactly once, and Data Sync does not claim that.

Why the checkpoint rule matters

The ordering of those two operations — flush, then checkpoint — is the whole thing. Consider the alternative.

If the checkpoint advanced when a batch was handed to the sink rather than when the sink durably acknowledged it, then a crash in the window between those two moments would leave a checkpoint pointing past changes that never landed. On restart, Data Sync would resume from the newer position and those changes would be silently and permanently lost. Nothing would report an error. The sync would show as healthy. You would find out weeks later from a row count that does not reconcile.

Flushing first inverts the cost of the crash. The window between the flush and the checkpoint report becomes a window of duplicates, not of gaps — and duplicates are exactly what an idempotent sink is built to absorb. Data Sync trades a recoverable problem for an unrecoverable one, in the direction that favours correctness.

The control-plane checkpoint is the source of truth for recovery. Runners report checkpoints upward on a debounce — at most one report every couple of seconds, or every ten thousand advanced rows, whichever comes first — and always flush a pending checkpoint before reporting a terminal status. A different runner can therefore pick up the same sync from the control-plane position, which is what makes a dead runner survivable rather than fatal.

The replay window is bounded, not zero. Its size is roughly the work done since the last reported checkpoint: at most a couple of seconds of changes, or ten thousand rows. That is the volume of already-applied changes your sink will be asked to absorb after an unclean restart.

Your guarantee is your sink's guarantee

Because the mechanism is replay into an idempotent sink, the strength of the guarantee is a property of the sink, not of Data Sync in the abstract. Data Sync classifies each sink by how it achieves idempotency.

Class	Mechanism	Depends on	Replay-safe at any time?
`upsert`	Writes are keyed insert-or-update; deletes are keyed deletes. Re-applying converges on the same row.	A stable primary key on the target.	Yes
`sequence_dedup`	Each row carries a monotonic change-sequence number; the sink durably discards anything at or below what it has already applied.	The sink storing the sequence durably.	Yes — regardless of how long the outage lasted.
`windowed_dedup`	Duplicates are suppressed by a dedup key held for a bounded period.	The replay falling inside that window.	Only inside the window. A longer outage replays duplicates that the sink no longer recognises.
`file`	Deterministic file naming — a replayed range writes to the same object path.	The consumer reading whole objects rather than tailing appends.	Partially. A replayed range re-appends.
`append_only`	None. Every event is appended as it arrives.	—	No. At-least-once by construction.

Mapping that onto the sinks these docs cover:

Sink	Class	Notes
PostgreSQL	`upsert`	`INSERT … ON CONFLICT (<pk>) DO UPDATE`, with deletes as `DELETE … WHERE <pk> = …`. Requires a stable primary key — see the warning below.
Cloud Spanner	`upsert`	Mutations API: `InsertOrUpdate` for inserts and updates, keyed `Delete` for deletes. Both are naturally idempotent.
Google BigQuery	`sequence_dedup`	Rows carry `_change_type` and `_change_sequence_number` (the source log position). Data Sync additionally filters out events at or below the last flushed position, so a replay after a restart does not re-append. Applying changes rather than appending them requires the target to declare `PRIMARY KEY (…) NOT ENFORCED`.
Kafka, Pub/Sub and other append-only targets	`append_only`	At-least-once by construction. There is no key to converge on and nothing to dedup against. A replay after a crash publishes messages the consumer has already seen. Dedup downstream, on a key you extract from the payload.

A target with no primary key degrades to append. Without a key there is nothing to upsert against, so replayed changes are inserted again and replays duplicate rows. This is the single most common way an exactly-once expectation gets broken in practice — the pipeline is configured correctly, the sink is an upsert-class sink, and the target simply has no key. Check the primary key on every target before you rely on the guarantee. On the Postgres sink specifically, a keyless target falls back to ON CONFLICT DO NOTHING so replays do not hard-fail, which additionally means updates are silently dropped for that table.

Ordering

CDC is strict FIFO per sink. A single writer applies changes in the order they were decoded from the source log, so per-table and per-key order is preserved: if a row is inserted, updated and then deleted at the source, the sink sees those three operations in that order. There is no reordering, no parallel-writer race on the same key, and no need for you to sequence anything yourself.

Snapshot is different, and deliberately so. Snapshot partitions are non-overlapping — each covers a distinct key or page range and no row belongs to two partitions — so there is nothing for an ordering rule to protect. Partitions therefore run fully concurrently, which is what makes a large backfill finish in reasonable time. Parallelism is horizontal across runners: one partition is one task.

Within a CDC batch, Data Sync applies compaction: last-event-wins per key. If a row changes five times inside one micro-batch, one write carries the final state rather than five writes carrying each intermediate state. This preserves final-state semantics — the row at the sink ends up correct — but it means the sink does not observe every intermediate value. If your downstream depends on seeing each individual revision rather than the settled state, a CDC replication sink is the wrong shape for that requirement.

Transactions

Changes originating from a single source transaction arrive as a contiguous run sharing one commit position. Nothing from another transaction is interleaved into that run.

That gives you a usable transaction boundary at the sink: a change in the commit position marks the end of one transaction and the start of the next. The commit position is available to your mappings as _meta.lsn, and the transaction identifier as _meta.txn_id, so you can materialise either as a derived column if you need to reconstruct transaction grouping downstream.

What this does not give you is atomic application of the transaction at the sink. The run is contiguous, but it is written as ordinary batched writes; a crash partway through can leave part of a source transaction applied and part not, until the replay completes. See What Data Sync does not promise.

Snapshot-to-CDC overlap

The handoff between the backfill and the stream is the classic place for a CDC pipeline to lose data, so it is worth restating exactly how Data Sync orders it.

The source log position is captured and persisted before the first partition is scanned. Only then does the backfill begin. When the last partition finishes, the CDC phase is dispatched starting from that captured position — not from the position at the moment the snapshot ended.

The consequence is that nothing committed during the backfill is lost. Every change that happened while the snapshot was running is still ahead of the captured position, so CDC replays it. Some of those changes were also picked up by the snapshot itself, because a partition scanned late in the backfill already reflects them. That overlap is real, and it is absorbed by idempotent writes in exactly the same way a crash replay is.

Persisting the position before scanning has a second benefit: a crash mid-snapshot still leaves a valid resume point, so a failed backfill does not invalidate the handoff.

Backpressure and throughput

Data Sync micro-batches. A batch flushes on whichever comes first:

Trigger	Threshold
Event count	1,000 events
Batch size	4 MiB
Time	100 ms
Source goes quiet	Immediate flush

The quiet-source flush is what keeps latency low on low-traffic streams: a single change on an otherwise idle table is not held waiting for 999 companions. Per-sink batch hints can override the byte ceiling — BigQuery advises roughly 500 rows or 5 MiB per request, sitting under the Storage Write API's own limit.

Backpressure is implicit and end-to-end. A slow sink fills the bounded write channel, which blocks the batcher, which blocks the reader, which stops pulling from the source cursor. There is no unbounded buffer anywhere in that chain and nothing to tune to make one appear — the pipeline runs at the rate the sink can absorb, and the memory footprint stays flat regardless of how far behind it falls.

With multiple sinks, the slowest one sets the overall rate. That is deliberate, not an oversight. Letting a fast sink run ahead would mean holding an unbounded amount of undelivered data for the slow one, or advancing the checkpoint past changes one sink had not durably taken. Both break the guarantee. If a fast target should not be held hostage by a slow one, give them separate syncs.

Optional rate limiting is available as a maximum events per second and a maximum bytes per second, when you want to cap the load Data Sync places on a source or a sink rather than letting it run as fast as both allow.

What you observe when a sink is the bottleneck is rising lag. The two meanings of that word are distinguished in Monitoring & runs.

What Data Sync does not promise

Stated plainly, so you do not design around something that is not there.

No cross-table transactional consistency at the sink. A source transaction touching three tables arrives as a contiguous run, but it is applied as ordinary batched writes. A reader querying the sink mid-batch can see one table updated and another not yet. If your downstream needs a point-in-time consistent view across tables, build that from the commit position — do not assume the sink is transactionally consistent at every instant.

No ordering between sinks. Ordering is FIFO per sink. If one sync fans out to two sinks, they progress independently and there is no guarantee about their relative position at any given moment. One can be seconds ahead of the other. Do not build a cross-sink comparison that assumes they are in step.

Deletes are not captured by poll-based sources. Elasticsearch is the case in these docs: its CDC mode polls a monotonically increasing watermark field rather than reading a change log, and a query cannot see a document that no longer exists. Deleted documents remain at the sink indefinitely. This is a property of polling, not a bug to be fixed — if you need deletes from Elasticsearch, model them as soft deletes with a watermark update.

Before-image completeness depends on your source configuration. Data Sync forwards what the source gives it, and sources differ:

Source	Default before-image	To get more
PostgreSQL	With the default `REPLICA IDENTITY`, `UPDATE` and `DELETE` before-images carry primary-key columns only.	`ALTER TABLE <t> REPLICA IDENTITY FULL;`
Cloud Spanner	With the default value capture type `NEW_ROW_AND_OLD_VALUES`, the after-image is the full row of all tracked columns but the before-image contains only the modified columns.	Enough for delete keys and audit; not a complete before-image. Configure the change stream's value capture type to suit.

This matters if you are building change auditing, computing deltas, or routing on a previous value. The after-image is complete; the before-image is only as complete as the source was configured to make it.

Before you rely on exactly-once, check three things. Does every target have a stable primary key? Is your sink an upsert or sequence_dedup class sink rather than append-only? And does your downstream tolerate the last-event-wins compaction within a batch? If all three are yes, the guarantee holds as described. If any is no, this page tells you what you actually have.