Core concepts

Data Sync has a small vocabulary, and every other page in these docs leans on it. This page defines each term precisely: what a sync is made of, what a run is, what a checkpoint means, which states a sync can be in, and where the work actually executes.

Connector, connection, sync

Three levels, each one a configuration of the level above it.

A connector's role decides where it can be used. A source-only connector never appears in the sink dropdown of the sync wizard, and vice versa. The console shows the role as a badge on every connector tile: Source, Sink or Source · Sink.

Why a connection holds a reference, not a secret

A connection stores a pointer to a credential — a Blunox-managed secret record, or a path in your own HashiCorp Vault, AWS Secrets Manager or GCP Secret Manager. It never stores the value. The reference is resolved at dispatch time, by the runner that is about to do the work.

That indirection buys three things. Rotating a password is a change in your secret store, not an edit to every connection and sync that uses it. Every resolution is a separately audited event, so credential access has a log independent of whoever edited the config. And for a bring-your-own store the value is read inside your own network and never reaches Blunox at all — which is what makes the BYOC deployment model meaningful rather than cosmetic. Security & deployment has the encryption details.

Replication modes

Set on step 1 of the sync wizard, and fixed once the sync has started.

Snapshot and CDC

These are the two phases of replication, and they work quite differently.

Snapshot is a parallel backfill. The control plane asks a runner to plan partitions for each table, persists that plan, and then dispatches one partition as one task. Parallelism is horizontal across the runner fleet rather than threads inside one process, which is why a large table's backfill scales by adding runners. Postgres partitions on a primary-key range (pk:<lo>-<hi>) or a CTID page range (ctid:<start>-<end>). Each partition reads from a consistent point-in-time view, and partitions are non-overlapping, so they need no ordering between them.

CDC is a single continuous stream. Changes are decoded from the source's own change log, mapped, batched and written. Ordering is strict FIFO per sink — one writer — because unlike a backfill, the order of changes to the same row is the whole point.

The handoff, and why the overlap is safe

The source log position is captured and persisted before any partition is scanned. When the last partition finishes, the CDC phase is dispatched starting from that captured position.

This means changes that commit at the source during the backfill are read twice: once by whichever partition happened to scan that row, and again by CDC replaying from the pre-backfill position. That overlap is deliberate. The alternative — capturing the position after the backfill — would silently lose every change committed while the snapshot ran. Data Sync chooses duplicates over gaps, because idempotent sink writes absorb duplicates and nothing absorbs a gap.

Capturing the position first also means a crash mid-snapshot still leaves a valid resume point. See Delivery guarantees for what "idempotent" means per sink — it is a property of the destination, not of Data Sync.

Runs

A run is one execution of a sync. There are two types, and they have fundamentally different shapes: a snapshot run has an end — it finishes when its last partition finishes — while a cdc run does not. A CDC run stays open for as long as the sync is streaming, which can be months. Do not read a CDC run's missing end time as a stuck run.

There is at most one active run per sync. A sync in running may show a completed snapshot run and an open CDC run at the same time; only the CDC one is active.

Inside a snapshot run, each partition carries its own status. The run detail page shows them in a table with the partition range, rows scanned against the estimate, and bytes.

Checkpoints

A checkpoint is the last source log position whose changes are durably written at the sink. It is a single number per sync, and it is the most consequential piece of state Data Sync keeps.

The rule that makes it useful is narrow and absolute: the checkpoint advances only after a sink flush durably succeeds. Not when the change is read from the log, not when it is mapped, not when it is handed to the sink client — after the sink has confirmed the write is durable. Everything else follows from that ordering.

Because the checkpoint lags the read position rather than leading it, a crash can only ever cause Data Sync to replay work it has already done, never to skip work it has not. On restart it resumes from the checkpoint, re-reads the window between the checkpoint and wherever it actually got to, and writes those changes again. The sink's idempotency absorbs them. This is the entire mechanism behind exactly-once delivery — there is no distributed transaction spanning the source read and the sink write, and Data Sync does not pretend otherwise.

The control-plane checkpoint is the source of truth, not the runner's local view. That is what lets a different runner pick up the same sync after the original one dies.

Reporting the checkpoint upward is debounced, because doing it on every advance would be a write per batch. A runner reports at most once every couple of seconds, or every ten thousand advanced rows, whichever comes first — and it always flushes a pending checkpoint before reporting a terminal status, so a clean stop does not leave the control plane behind. The practical consequence: the checkpoint the console shows can be a couple of seconds stale even on a perfectly healthy sync. That is the debounce, not lag.

Sync states

Ten states, and this is the complete set. There is no paused state, no archived state, and no generic error state.

The paths between them are constrained. A sync starts life in draft and moves to created when its definition is complete; created is the only state from which a first starting is reachable. From starting, a sync lands in snapshotting or running depending on its mode, or goes straight to a terminal state if the dispatch itself fails. snapshotting and running flow into each other in the expected direction — backfill then stream — and either can move to pending_schema_approval, stopped, completed, terminated or failed.

All four non-terminal exits lead back to starting: stopped→starting on a manual restart, terminated→starting after a crash, failed→starting once you have fixed the cause, and pending_schema_approval→starting when the change is approved. A sync sitting in pending_schema_approval or failed can also simply be stopped. Only completed is genuinely terminal.

Two edges look odd until you know what they are for: running→starting and snapshotting→starting. These are the crash-recovery edges. When the control plane detects that the runner executing an apparently-live sync is gone, it moves the sync back to starting and re-dispatches it — from the control-plane checkpoint, possibly onto a different runner. A sync that briefly flips to starting without anyone touching it has been recovered, not restarted by a person.

Runners

A runner is the execution host: the process that opens a connection to your source, reads the change log, applies your mappings and writes to your sink. It is the data plane. The control plane — the console, the sync definitions, the scheduler — never touches your rows.

This split is the single most important thing to understand about the architecture, because it decides which deployment model you need. Rows travel source → runner → sink. What the control plane stores is metadata: sync definitions, checkpoints, run history, table and column names, and operational metrics like lag and throughput. Row values never traverse it. Run logs are held to the same rule — log attributes carry metadata only, never row data.

Each runner declares the capabilities it can serve — pulse.cdc.postgres, pulse.cdc.sqlserver, mira.pushdown and so on — and is bound to an environment. Work is only dispatched to a runner whose capabilities and scope match.

A BYOC runner holds one long-lived outbound connection on TCP 443 with a 15-second heartbeat. There is no inbound path, no public IP and no database of its own — which means there is nothing to expose in your network and nothing to firewall open.

Two meanings of lag

The word appears in two places in the console and means two different things. Confusing them leads to chasing the wrong problem.

Stream lag is the operational number — it is what the Monitor derives its lagging status from, and what an alert threshold is measured against. Just remember that on a quiet table it drifts upward for a benign reason. A steady climb under active write traffic is the signal worth acting on. Monitoring & runs covers how status is derived and how to set thresholds that do not page you for silence.