Monitoring & runs
Once a sync is streaming, the questions change. Not "did it start" but "is it keeping up", "which table is holding up the backfill", and "why did run 7 fail at 02:40". This page is the operator's map: which screen answers which question, how status and lag are actually computed, and what the logs, alerts and metrics will and will not tell you.
The three views
Data Sync gives you three levels of zoom, and it is worth knowing which one you want before you start clicking.
Syncs is inventory. Every sync in the workspace, one row each, with its state and lag. Use it to answer "what exists and is any of it unhappy".
Monitor is the live fleet. Same population, sorted worst-first, with incidents, SLA and acknowledgement on top. Use it when you are on call and want the one thing that needs you now.
Sync detail and run detail are diagnosis. A single sync's checkpoint, its run history, and — inside a run — the per-partition breakdown and the log lines. Use these once you know which sync is the problem.
Syncs list
Syncs is the inventory view. It summarises the workspace in four figures — streaming now, rows/min, average lag and tables replicated — and lists every sync with its state and lag.
The state shown for each sync is its real lifecycle state, not a derived health word: a sync that is still backfilling reads snapshotting, not lagging.
| Column | What it shows |
|---|---|
| Source → sink | The two connections the sync binds together. |
| Type | The replication mode: snapshot, cdc or snapshot_then_cdc. |
| Tables | How many tables the sync replicates. |
| Lag | Stream lag — how stale the committed checkpoint is. See Two meanings of lag. |
| State | The sync state: draft, created, starting, snapshotting, running, pending_schema_approval, stopped, completed, terminated or failed. |
Sync detail
Opening a sync shows four summary figures and two tables.
| Tile | Reading it |
|---|---|
| State | The current lifecycle state. running means CDC is live; snapshotting means the backfill is still in flight. |
| Last run | The most recent run and how it ended. A sync in running can show a completed snapshot run alongside an open CDC run — there is at most one active run at a time. |
| Checkpoint | The last source log position durably written at the sink, and how far behind that position is now. This is the number that matters for recovery: a restart resumes here. |
| Tables | How many tables this sync replicates. |
The run history table lists every execution of the sync: run number, type (snapshot or cdc), started, ended, duration, events, bytes and status. Statuses are the run states — starting, running, stopped, failed, terminated, completed.
Run history is append-only. A sync that has been stopped and restarted several times shows every attempt, which is usually the fastest way to spot a crash loop.
Below it, the replicated tables list shows each source table, the target it writes to, and how its columns are treated — which are included, renamed or derived. It is the quickest way to confirm that a mapping change actually took effect, since mappings are compiled when the sync starts rather than when you edit them.
Run detail
A run page opens with status, duration, events, bytes and table count. For a CDC run that is usually all you need. For a snapshot, the interesting part is further down.
Data Sync does not backfill a table with one long scan. The control plane asks a runner to plan partitions for each table, persists the plan, then dispatches one partition as one task. Parallelism is horizontal across runners rather than threads inside one process, and partitions are non-overlapping, so they need no ordering between them. The consequence for you as an operator is direct: a slow backfill is almost never uniformly slow. It is one partition on one table, and the partition table is where you see it.
If a snapshot seems stuck, look for a partition still running while the others have completed. That one range is what the snapshot phase is waiting on.
| Partition column | What it tells you |
|---|---|
| Table | The source table this partition belongs to. One table produces many partitions. |
| Partition range | The bounds the partition scans. Postgres partitions on a primary-key range (pk:<lo>-<hi>) or a CTID page range (ctid:<start>-<end>). |
| Status | pending, running, completed, failed, skipped or terminated. |
| Rows scanned / estimate | Progress against the planner's row estimate. A wildly low ratio on one partition usually means a skewed key range or a missing index on the partitioning column. |
| Bytes | Bytes read for that partition. Compare against rows to spot a table whose average row is far larger than the rest. |
Partitions target roughly 100,000 rows or 2 GiB and are capped at 512 per table, so a table split into far fewer partitions than you expected is a sign the planner could not find a usable range. A partition that failed is re-dispatched automatically with bounded retries (three by default); one that stays failed after that is a real error and its cause is in the run's log lines.
Monitor
Monitor is the live fleet view. It summarises the fleet in five figures: fleet health, lagging, errored, throughput and p95 lag. The p95 figure is the one to watch over time — an average lag hides a single sync that is minutes behind, a p95 does not.
Incidents are acknowledged from the same view.
The sync list is sorted worst-first, so you do not have to hunt. Three controls narrow it further:
- Needs attention — filters to syncs that are not
healthy. On call, this is usually the only view you want. - Grouping — by environment, source or owner. Grouping by source is how you tell one bad database apart from a general problem; grouping by environment keeps a noisy staging workspace out of your production picture.
- Incidents panel — open incidents with acknowledgement. Acknowledging records that a human has seen it, so the row stops competing for attention without hiding the underlying condition.
Status, and how it is derived
Monitor does not show sync states directly. It collapses state and lag into one of four status words, evaluated in order — the first matching rule wins.
| Status | Derived when |
|---|---|
error | The sync has failed. |
paused | The sync is stopped, terminated, draft or created — that is, not currently doing work, whether by your choice or after a crash. |
lagging | The sync is waiting on a schema approval (pending_schema_approval), or its lag is at least two seconds. |
healthy | Everything else. |
The two-second threshold is deliberately tight. Data Sync micro-batches flush on whichever comes first — 1,000 events, 4 MiB or 100 ms — and flush immediately when the source goes quiet, so a healthy stream sits well under a second. Crossing two seconds means the pipeline is genuinely behind rather than merely between batches, and it is early enough to be a warning rather than an incident. If your workload legitimately runs hotter than that, set the alert threshold you actually care about rather than reading the Monitor colour as an outage.
paused covers terminated: a crashed sync and a stopped one are both "not moving data right now", and both are restartable. Use the state column on the Syncs list or the sync detail page when you need to tell them apart.Row actions
Each Monitor row carries four actions.
Actions apply to a single sync. There is no bulk apply: a re-snapshot across a fleet is a deliberate, one-at-a-time decision.
| Action | What it does, and when you would use it |
|---|---|
| Pause | Stops the sync, moving it to stopped. The checkpoint is preserved, so restarting resumes from the last durably written position rather than re-reading. Use it when the source or sink needs maintenance and you would rather not accumulate errors during the window. |
| Re-snapshot | Re-runs the backfill for the sync. This is the recovery tool when the target has drifted from the source — rows deleted at the sink out of band, a target truncated by mistake, or a period where you are no longer sure the stream was complete. Because sink writes are idempotent, re-reading rows that are already present converges rather than duplicating. |
| Acknowledge | Marks the incident on that row as seen. It changes nothing about the sync; it changes what the on-call view emphasises. Use it once someone owns the problem, so the next person looking at Monitor is not drawn to the same row. |
| Edit SLA | Adjusts the lag expectation this sync is judged against. Use it when a sync's normal behaviour is not the fleet default — a low-traffic table that batches infrequently should not sit permanently in needs attention. |
Two meanings of lag
"Lag" appears in more than one place in Data Sync and does not mean the same thing in both. The distinction matters when the two disagree.
| Kind | Definition | Where you see it |
|---|---|---|
| Event lag | The time between a change committing at the source and Data Sync processing it. | Per-event processing latency — the pipeline's own responsiveness. |
| Stream lag | How stale the committed checkpoint is: now minus the time the checkpoint last advanced. | The Lag column on the Syncs list, the Monitor tiles and rows, and the status derivation above. |
The lists show stream lag. That is the conservative number, because the checkpoint only advances after a sink flush durably succeeds — it reflects what is safely written, not what has merely been read. It also means stream lag rises on a quiet source: if nothing has committed for five minutes, nothing has advanced the checkpoint for five minutes either. Runners report checkpoints on a debounce (at most one report every couple of seconds, or every ten thousand advanced rows, whichever comes first), which puts a small floor under the reported figure.
Run logs
Logs are scoped to a run, not to a sync — open the run, and the log panel is on the same page as its partition table. Two filters:
| Filter | Values |
|---|---|
| Severity | DEBUG, INFO, WARN, ERROR. Producers emit INFO and above, so DEBUG is present in the filter but sparsely populated in practice. |
| Source | engine, runner, gateway, control-plane. The tag tells you which layer spoke: a connection error tagged runner is a network or credential problem near your data, one tagged control-plane is orchestration. |
A live tail is available while a run is open, which is what you want during a start-up or a re-snapshot.
Log volume is bounded on purpose, and the bounds are worth knowing before you rely on logs for a post-mortem:
- 10,000 lines per run. After that, only errors are retained and a truncation marker is written into the stream so you can see where the cut happened.
- 30-day retention by default, configurable per workspace.
ERROR.Alerts
Alerts are configured per sync. Three events can fire one:
| Event | Fires when |
|---|---|
sync_error | The sync errors. |
lag_high | Lag exceeds a threshold you set, in seconds. Set this from the sync's own normal behaviour, not from the Monitor's two-second status threshold. |
schema_drift | Data Sync observes a structural change on a replicated table. See Schema changes for what is classified as breaking. |
Every alert has a sustain duration: the condition must hold continuously for that long before the alert fires. This is what keeps a single slow batch or a momentary reconnect from paging anyone. Set it longer than the longest normal blip you are willing to tolerate.
Delivery goes to Slack, PagerDuty or a named webhook. After an alert fires, a silence window suppresses repeat notifications for that alert, so a sync flapping in and out of a threshold produces one page rather than fifty.
Metrics
Data Sync exports its operational telemetry over OpenTelemetry, so the console is not the only place this data can live. Point your collector at it and the same signals land in whatever you already run.
The exported families cover:
- Replication lag and throughput per sync.
- Event and byte counts on both sides — what was read from the source and what was written to each sink. Comparing the two is how you find a sink that is dropping behind the reader.
- Snapshot progress — partition and row progress through a backfill.
- Batch duration — how long a micro-batch takes end to end, which is where sink slowness shows up first.
- Checkpoint counts — checkpoint advances, the durability signal behind stream lag.
- Schema-drift counts — structural changes observed, by classification.
These carry the same privacy property as logs: metric attributes identify syncs, tables and sinks, never row values.