Cloud Bigtable
Data Sync reads Google Cloud Bigtable as a source: it backfills the tables you select, then streams changes from each table's change stream. There is no Bigtable sink.
Overview
A Bigtable change stream records every mutation to a table. Data Sync reads it rather than re-scanning the table, so each change is seen once. The stream is split into partitions that change as the table splits and merges, which is what the heartbeat and partition-retry settings under Advanced settings tune.
Before you connect
- Enable a change stream on every table you want to replicate, with a retention period long enough to cover an outage. If Data Sync is stopped for longer than the retention period, the sync cannot pick up where it stopped.
- Grant access to the identity Data Sync uses: it needs to read the tables and their change streams.
Connection fields
Set once, on the connection, in step 2 of the add-a-connection wizard.
| Field | Default | Description | |
|---|---|---|---|
project_id | Required | — | The Google Cloud project that owns the instance. |
instance_id | Required | — | The Bigtable instance. |
app_profile_id | Optional | The instance's default profile | The app profile Data Sync's reads go through, so you can route them to a specific cluster. |
endpoint | Optional | Google Cloud | A custom endpoint. Leave it empty for the standard endpoint. |
credentials_file | Advanced | — | A service-account key file for the runner to use. Most connections attach the key in step 3 instead. |
heartbeat_interval | Advanced | 5s | How often the change stream reports progress on a quiet partition. |
partition_retry_backoff | Advanced | 1s | The wait before retrying a change-stream partition that failed. |
Credentials
Step 3 asks for Credentials JSON: a service-account key, stored as a Blunox-managed secret or as a reference to your own secret store. Leave it empty when the runner already has an identity in the project. See Connections.