Quickstart

This walks you from an empty workspace to a sync that streams a PostgreSQL database into BigQuery: one source connection, one sink connection, and a five-step wizard. Everything happens in the console — there is nothing to install and no configuration file to write.

The example values below are placeholders. Use your own hosts, projects and names wherever you see them.

1. Prepare PostgreSQL

Data Sync reads PostgreSQL's own change log through logical decoding, which is off by default. Do this on the server first: a connection can test green and still be unable to stream if it is missing.

1. Turn on logical decoding. Set wal_level = logical and restart the server. It is a restart, not a reload.
2. Leave room for Data Sync. Each sync uses one replication slot and one WAL sender, so make sure max_replication_slots and max_wal_senders have one free of each.
3. Create a role for Data Sync with the REPLICATION attribute and read access to the tables. The snapshot reads them as ordinary queries.
4. Optionally, pre-create the publication with exactly the tables you are willing to expose. Data Sync uses an existing publication as-is and never widens it.

Every prerequisite, including REPLICA IDENTITY and when you need it, is on the PostgreSQL connector page.

2. Prepare BigQuery

1. Create the dataset Data Sync will write into, in the location you want the data to live. Data Sync creates the tables inside it for you.
2. Decide how Data Sync authenticates. Either create a service account with BigQuery Data Editor on the dataset and download its JSON key, or, if your sync will run on a BYOC runner that already has an identity in the project, skip the key and let it use that identity.

3. Add the PostgreSQL connection

A connection is created once and reused by every sync that points at the same database. Open Connections and choose Add connection. The wizard has four steps.

1. Connector — choose PostgreSQL. Each connector carries a role badge: Source, Sink or both. PostgreSQL is Source · Sink, so it can sit on either side of a sync. The badge comes from the connector itself, and it decides where the connection can be picked later.
2. Configure — name the connection shop-postgres, then enter the host db.example.com, port 5432, database shop and username pulse. Leave SSL mode on require, or choose verify-full if you have a CA to validate against. If you pre-created a publication in step 1, open Advanced settings and enter its name, pulse_pub_shop, under Publication; otherwise leave the advanced settings at their defaults.
3. Credentials — supply the role's password. Choose where it lives (see the table below). The connection stores a reference to the secret, never the value.
4. Test & save — run the test. It runs from a runner in the connection's environment, so a pass confirms the network path the sync will actually use, the SSL mode and the login. Save only once it passes.

You can switch a connection between the two later, but it re-points every sync that uses it, so choose per environment up front.

If the test fails, it is almost always one of three things: a firewall that has not been told about the static egress IPs for your workspace region, an SSL mode the server will not accept, or a role without the privileges from step 1. The test does not check that logical decoding is configured. That is checked by the connector's preflight when the sync starts.

4. Add the BigQuery connection

Same wizard, same four steps.

1. Connector — choose BigQuery. It is Sink only, so it will appear when you pick a sink and never as a source.
2. Configure — name it analytics-bigquery and enter the project example-analytics and dataset shop_replica. Those are the only two required fields. Partitioning and staleness sit under Advanced settings; by default, tables Data Sync creates are partitioned by day.
3. Credentials — paste the service-account key into Credentials JSON as a Blunox-managed secret, or reference it in your own secret store. Leave it empty to use the runner's own identity.
4. Test & save — run the test and save once it passes.

BigQuery is a true CDC sink: an update at the source overwrites its row and a delete removes it, as long as the target table has a primary key. Data Sync declares the key when it creates the table, from the keys set on the sync. The details are on the BigQuery connector page.

5. Create the sync

Open Syncs and start the create wizard. It has five steps.

1. Basics — name the sync shop-to-bigquery and leave the mode on Snapshot, then CDC (snapshot_then_cdc). It backfills everything in the tables today, then keeps streaming, with the hand-off handled so nothing committed during the backfill is lost. Snapshot only is a one-time copy that stops when it finishes. CDC only streams changes without a backfill and assumes the rows already exist at the sink. The mode is fixed for the life of the sync; to change it, create a new sync.
2. Connections — pick shop-postgres as the source and analytics-bigquery as the sink. Only connections whose role allows it appear in each list. Check you have the pair you meant: nothing stops you pointing a production source at a staging sink.
3. Tables — Data Sync reads the source schema and lists its tables. Select public.orders and public.order_items. A table you do not select is never read, so this is also the cheapest place to control load on the source.
4. Mappings — the defaults are right for a first sync: every column included, names unchanged, and each table's primary key taken from the source, which is what lets BigQuery apply updates and deletes. The targets default to orders and order_items in shop_replica. Renames, excluded columns and derived columns are covered in Tables & columns.
5. Review — check the summary. It is the last place the whole configuration appears together; after creation it is spread across the sync's detail page.

6. Test it before you start it

On the Review step, choose Test this sync. The dry run samples real rows from orders and order_items, pushes them through the exact mappings this sync would use, and shows you the output: the column names, types and values BigQuery would receive.

Nothing is written to BigQuery. The sampled rows are discarded rather than stored, and the dry run is audited separately from a real run. It is the cheapest way to catch a wrong primary key or a derived column that returns something you did not expect.

Then finish the wizard. Create & start creates the sync and starts it immediately. Create draft saves it as a draft you can come back to; a sync can be edited in place only while it is draft or created.

7. Start it and watch

After Create & start, the sync moves through three states:

1. starting — credentials are resolved and the PostgreSQL preflight checks logical decoding. A failed preflight stops the run here rather than letting it half-start.
2. snapshotting — the backfill runs in partitions across the runners.
3. running — the backfill is done and CDC streams from the log position captured before it began.

Open the sync to follow it. Its detail page shows the current state, the checkpoint position and how far behind it is, the run history, and each replicated table with its target. Open any run to see its partitions — the range each one covers, its status and rows read against the estimate — and that run's log.

The figure reading checkpoint … behind is stream lag: now minus the time the committed checkpoint last advanced. It measures how stale your durably written position is, not how long one row took to arrive. A healthy CDC sync on a quiet table can show a second or two simply because nothing has changed. A number that climbs steadily is the one to care about.

Once the sync is running, query the target to check the rows arrived. Every table Data Sync writes starts with two metadata columns, _pulse_lsn and _pulse_op; the data columns follow, sorted by name, so query them by name rather than position.

What next