Cloud Spanner

The spanner connector works in both directions. As a source it reads a Spanner change stream through a partition tree that splits and merges with the database underneath it. As a sink it writes with the Mutations API, which is idempotent by construction and therefore replay-safe.

Overview

Spanner does not have a write-ahead log you can tail. What it has instead is change streams: a schema object you create with DDL, which watches some or all of the database and exposes committed changes through a partitioned read API. That is the capture mechanism on the source side, and it is why setting up a Spanner source involves a DDL statement rather than a server setting.

On the sink side, Data Sync writes mutations rather than DML. A mutation is a key-addressed instruction — write this row, delete this key — which Spanner applies without a read-modify-write cycle. Two properties follow from that and shape everything else on this page: mutation writes are naturally idempotent, and Spanner meters them against a per-commit ceiling that counts columns, not rows.

The same connector serves both roles, and one connection can be the source of one sync and the sink of another. The advanced fields on the connection are shared, so a few of them only matter in one direction.

As a source

How it captures changes

A change stream is read through a partition tree, not a single cursor. Spanner splits and merges the underlying data as it rebalances, and the change stream reflects that: a partition can end and hand off to two children, or two partitions can merge into one. The set of partitions Data Sync is reading is therefore not fixed for the life of the sync.

Data Sync persists a watermark per partition — the timestamp up to which that partition has been fully consumed. When a partition ends and names its children, those children are not scheduled immediately. They are scheduled only once the minimum watermark across all live partitions has advanced past their start timestamp.

That rule is what preserves per-key ordering across a split. A key that moves from one partition to another during a rebalance would otherwise be at risk of having its newer change read before its older one; waiting for the minimum watermark guarantees every change before the split has already been emitted. It is also why a single stalled partition slows the whole stream — that is the cost of the ordering guarantee, and it is deliberate.

The snapshot phase is separate and does not use the change stream at all. It issues stale reads at a single consistent timestamp with key-range predicates, so every partition of the backfill sees exactly the same version of the database and the partitions can run concurrently without coordinating.

Setting up the change stream

You can point Data Sync at a change stream you already manage, or let it create one. Setting auto_create_stream on the sync issues this DDL:

CREATE CHANGE STREAM IF NOT EXISTS pulse_orders FOR ALL
  OPTIONS (value_capture_type = 'NEW_ROW_AND_OLD_VALUES', retention_period = '7d')

Creating the stream requires the spanner.databases.updateDdl permission on the database. If the principal Data Sync authenticates as does not have it, leave auto_create_stream off, create the stream yourself with the statement above, and set change_stream to its name.

Preflight then verifies two things before the sync starts: that the named change stream exists, and that it watches at least one table. A stream that exists but tracks nothing is a common outcome of a hand-written FOR clause, so it is checked explicitly and the run fails with a message naming the stream rather than silently producing no events.

The stream name must match ^[a-zA-Z][a-zA-Z0-9_]*$. If you do not set change_stream, it defaults to pulse_ followed by the sync name.

Value capture

The stream is created with value_capture_type = 'NEW_ROW_AND_OLD_VALUES'. The after-image is the full row across all tracked columns, so inserts and updates arrive complete.

Connection fields

Set once on the connection. The advanced block is shared with the sink role, so some of it only applies when this connection is used as a target.

The three identifiers address the database; everything under Advanced settings tunes the sink path. The service-account key is attached separately in step 3, so the connection stores a reference rather than the JSON.

Sync fields

Set per sync, on the source side of the wizard.

Partitioning

Snapshot partitioning depends on the shape of the primary key. On an INT64 key, Data Sync splits the range by minimum and maximum value. On a STRING key, it samples boundary keys and splits between them.

Type mapping

As a sink

How it writes

Every change becomes a Spanner mutation: InsertOrUpdate for inserts and updates, and a keyed Delete for deletes. Neither depends on the current state of the row, so applying the same mutation twice produces the same result as applying it once. That is where the idempotency comes from — Data Sync does not need a dedup table or a sequence column on this sink.

Within a batch, events are grouped by target table and then hash-partitioned by primary key. Partitioning by key hash means every change for a given row lands in the same partition and stays in order relative to itself, while different keys are free to commit independently. Those partitions are committed concurrently.

Commit limits

A commit is split on whichever limit is reached first: max_mutations_per_batch (default 5000), or roughly 4 MiB of commit payload. The byte ceiling is there because a table with wide string or BYTES columns hits a size limit long before it hits a row count.

The mutation limit needs more care than it looks like it does, because Spanner counts mutations per column, not per row, and adds one more for every secondary-index entry the write touches. Its own hard ceiling is 80,000 mutations per commit.

So consider a table with 40 columns and three secondary indexes. Each row written costs roughly 40 column mutations plus 3 index mutations — call it 43. At the default max_mutations_per_batch of 5000 rows, that commit would carry about 215,000 mutations and Spanner would reject it. A table like that needs max_mutations_per_batch lowered to something closer to 1500 to stay under the ceiling with headroom. The narrower the table and the fewer the indexes, the higher you can leave it.

CPU throttling

Write concurrency backs off when the instance's CPU utilisation rises above cpu_throttle_threshold, which defaults to 0.7. This exists so a backfill into a Spanner instance that is also serving production traffic degrades gracefully instead of consuming the instance. Lower it if replication is competing with serving latency; raise it if the instance is dedicated to the load.

Table creation

Table creation is off by default — the target must already exist, and Data Sync issues no DDL. Turning on auto_create_table changes that in two ways.

If the target table is absent, Data Sync issues a CREATE TABLE … PRIMARY KEY (…) derived from the incoming Arrow schema and the key columns configured for that table mapping. If the target exists but is missing columns, Data Sync widens it with one ALTER TABLE … ADD COLUMN per missing column. Existing columns are never modified or dropped.

Added columns are always nullable. This is not a preference — Spanner cannot add a NOT NULL column to a populated table without a default value, so a nullable column is the only form the statement can take. Key columns are never altered, because changing a key would mean rewriting the table's physical layout.

Interleaved tables

Spanner enforces parent-existence for interleaved child rows: you cannot insert a child before its parent. During a backfill that is a real problem, because parent and child tables load as independent partitions with no ordering between them.

Arrow → Spanner DDL types

What auto_create_table writes when it creates or widens a target.

The row worth reading twice is Time. Spanner has no TIME type, so a source TIME column lands as STRING(MAX). Comparisons and ordering on that column at the sink are lexicographic string operations, not time arithmetic. If you need it to be a real time value downstream, combine it with a date at the source or cast it in the consuming query.

Delivery guarantee

Because InsertOrUpdate and keyed Delete are both idempotent, this sink is in the upsert idempotency class: a replayed range converges on the same state no matter how long the outage lasted, provided the table has a stable primary key. That is the strongest class Data Sync offers, and it is what makes a Spanner target safe to restart after a crash without reasoning about how far the checkpoint rolled back. The per-sink matrix is in Delivery guarantees.

Spanner Omni

Spanner Omni is the self-hosted variant, and Data Sync exposes it as a separate connector — spanner-omni — with the same implementation behind it. Everything on this page applies unchanged: the same change-stream capture, the same mutation writes, the same fields and type mappings.

Two differences, both in how you address it. endpoint is required rather than optional, because there is no Google-hosted address to fall back to. And there is no credentials_file — Omni connects without GCP service-account credentials, so the credential step of the connection wizard has nothing to attach.