Creating a sync

A sync is one source connection, one sink connection, a set of tables and a replication mode. You build one in a five-step wizard: Basics, Connections, Tables, Mappings, Review. Nothing moves until you start it, and you can dry-run the whole thing first.

Before you begin. You need at least one source connection and one sink connection already saved. If you have not created them yet, do that first — see Connections. You can also add one mid-wizard without losing your place.

Step 1 · Basics

Give the sync a name and pick how it replicates. The name is what you will look for in the Syncs list and in Monitor at three in the morning, so name it after what it does — the source and the dataset — rather than after the ticket that asked for it. The description is optional and shows on the sync detail page.

The replication mode decides which phases run.

Mode	Value	What runs	Choose it when
Snapshot only	`snapshot`	One full copy of every selected table, then the sync finishes.	You want a one-time migration or a periodic full reload. The sync ends in `completed`, which is terminal.
CDC only	`cdc`	Change streaming only. No backfill.	The target already holds the historical rows — you backfilled by other means, or you are re-pointing a stream at a table that is already in place.
Snapshot, then CDC recommended	`snapshot_then_cdc`	Full copy first, then continuous streaming from the log position captured before the copy started.	Almost always. This is the default, and the only mode that gets you from an empty target to a live one with no gap.

The reason snapshot_then_cdc is the default is the handoff. Data Sync captures the source log position before the first partition is scanned, so every change committed during the backfill is replayed once the copy finishes. Choosing cdc on an empty target does not backfill it, and choosing snapshot on a live table gives you a copy that starts going stale the moment it lands.

The mode you pick here is fixed for the life of the sync. To change it later, create a new sync rather than editing a running one.

Step 2 · Connections

Pick the source connection and the sink connection. Each picker only offers connections whose connector declares support for that role: a connector that is source-only never appears in the sink list, and a sink-only connector never appears in the source list. If a connection you expected is missing, the connector behind it does not support that direction.

Both pickers have an inline Add a connection escape hatch. It opens the connection wizard — 1 Connector → 2 Configure → 3 Credentials → 4 Test & save — and when you save, the new connection is selected and you are back where you were. You do not lose the name and mode you already entered.

The pair you choose here determines everything downstream: which tables can be discovered in step 3, which types can be mapped in step 4, and what delivery guarantee the sync can offer. Sink idempotency is a property of the sink connector, not of Data Sync — see Delivery guarantees.

Check the environment each connection belongs to before you continue. Nothing stops you pointing a production source at a staging sink.

Step 3 · Tables

Data Sync reads the schema from the source connection and lists what it finds: table names, row estimates and the primary key it detected for each table. Those three facts come from the source itself, not from anything you typed — the row estimate is what drives snapshot partitioning, and the detected key becomes the default primary key for the target unless you override it in step 4.

Selecting tables

The filter box narrows the list as you type; Select all applies to whatever the filter is currently showing, not to the whole database, which is the fast way to take an entire schema prefix without ticking eighty boxes. Only selected tables are replicated — a table with no rule is dropped, silently and by design.

Target names and create mode

Each selected table gets a target name, defaulting to the source table's bare name. Rename it here if your sink uses a different convention. Each table also gets a create mode:

Create mode	Behaviour
Auto-create	Data Sync creates the target object at the sink from the mapped schema.
Use existing	Data Sync validates the target that is already there and never issues DDL. Use this when the target is managed by your own migrations.

Combining tables

Combine into one table folds several selected source tables into a single target — the usual case is a sharded or partitioned source, events_2024, events_2025, and so on. Every member of the combined set carries a _source_table provenance column, and that column is forced into the target's primary key so that rows sharing an id across two source tables cannot collide. The full model is in Tables & columns.

If discovery cannot reach the source — a credential that has not been granted read on the catalog, a network path that is not open yet — the list comes back empty. You can still add tables by name and continue building the sync. Column-level mapping in step 4 needs a discovered schema, so a sync built this way maps every column straight through until the schema is read at start.

Row estimates come from the source’s own statistics. Data Sync uses them to plan snapshot partitions, so a badly stale estimate produces a lopsided but still correct backfill.

Step 4 · Mappings

The default here is 1:1 — every discovered column is included and keeps its name. You only touch this step to configure exceptions: uncheck a column to drop it before it leaves the source, type a new name to rename it, override the primary key for a target, add a calculated field whose value is an expression, or add a metadata column such as the commit time or the change operation.

Each table also carries its own schema-change policy here, which decides what happens when a column appears or changes type mid-stream.

This step has enough depth to deserve its own page. See Tables & columns for column selection, renaming, primary keys, derived columns and the full function catalog.

You do not have to change anything on this step: a sync with no mapping changes is valid.

Step 5 · Review

The review step is a single summary table: name and mode, source and sink, how many tables are selected, and what will happen to each one — its target, its create mode, and whether any columns are dropped, renamed or derived. Read the table count and the mode line carefully, because those are the two things you cannot change after the sync starts.

From here there are two exits.

Exit	Resulting state	What happens
Create draft	`draft`	The definition is saved and nothing is dispatched. Come back and edit it as often as you like.
Create & start	`created` → `starting`	The sync is created and immediately started. Credentials are resolved at dispatch time and any connector preflight check runs before the first read.

The review is the last place the whole configuration appears together. After creation it is spread across the sync’s detail page.

Testing before you start

The review step offers a dry run. It samples real rows from the source, runs them through the exact mapping rules this sync would use, and shows you the transformed output — renamed columns, dropped columns, derived values, the lot.

Nothing is written to the sink. No target is created, no DDL is issued, no row is delivered. The sampled rows are streamed through the mapping layer and discarded; they are never persisted anywhere in Data Sync. The dry run is audited separately from a real run, so an auditor can tell a test apart from a replication.

Use it to answer the questions that are expensive to answer later: does that expression compile against the real schema, is the primary key you overrode actually present in the output, does the renamed column land where the downstream query expects it.

What a dry run does not prove. It exercises the read path and the mapping layer, not the write path. It will not surface a sink permission that is missing, a quota, or a target whose existing schema conflicts with Use existing. Those show up on the first real run.

Starting, stopping, restarting

Starting moves the sync to starting. Credentials are resolved as a reference at dispatch time rather than being stored on the sync, and connectors that implement a preflight check run it now — SQL Server, Elasticsearch and Spanner all do, and a failed preflight blocks the run instead of letting it fail halfway through a backfill.

From there the mode decides the path. With a snapshot phase the sync goes to snapshotting: the control plane plans partitions per table, persists the plan, and dispatches one partition as one task, so parallelism is horizontal across runners. When the last partition finishes, CDC picks up from the log position captured before the backfill began and the sync settles into running.

A sync can only be edited while it is draft or created. Once it has started, edits are rejected — including the table list, the mappings and the mode. To change a running sync's shape, stop it and create a new one. This is deliberate: the checkpoint is only meaningful against the mapping rules that produced it.

Stopping and restarting

Stopping moves the sync to stopped. A crash lands it in terminated and a non-recoverable error in failed; all three can be restarted. Restarting resumes from the last durable checkpoint rather than starting over — the checkpoint only ever advances after a sink flush has durably succeeded, so the worst case on restart is that a bounded window of already-applied changes is replayed and absorbed by the idempotent sink.

completed is the exception: it is terminal, and it is what a snapshot-mode sync reaches when its one copy has finished.

Re-snapshotting

If a target has drifted — someone truncated it, a schema change was mishandled, a downstream migration rewrote it — you can re-snapshot from the row actions in Monitor. The same row menu also carries Pause, Acknowledge and Edit SLA.

A re-snapshot re-runs the backfill for the sync’s tables. Budget for the read load on the source before you trigger it on a large table during business hours.

Draft versus created

The two are easy to confuse because both mean "not running yet".

State	Meaning	Next state
`draft`	Saved from the wizard without starting. Incomplete definitions are allowed — this is the state you park a half-finished sync in.	`created`
`created`	A complete, valid definition that has never been started. This is what Create & start passes through on its way to `starting`.	`starting`

Both are editable and neither has a checkpoint, a run history or a runner attached. In Monitor both are reported as paused, alongside stopped and terminated — the fleet view cares about "not moving", not about why.