Google Cloud Pub/Sub
Google Cloud Pub/Sub can sit on either side of a sync. As a sink, Data Sync publishes each change as a message to a topic. As a source, it reads messages from a subscription. As a sink it is append-only, so delivery is at-least-once.
Overview
Pub/Sub fans changes out to any number of subscribers, each reading at its own pace. When a runner restarts and replays changes from its last checkpoint, they are published again, so a subscriber can see the same change twice. That is why Pub/Sub is append_only in the delivery guarantees matrix. Design subscribers to be idempotent.
By default Pub/Sub does not keep messages in order. If your subscribers need changes to the same row in sequence, turn on Enable message ordering: messages that share an ordering key are then delivered in the order they were published, to subscriptions that also have ordering enabled.
Before you connect
- Create the topics and subscriptions the sync will use.
- Grant access to the identity Data Sync uses: publish rights on the topics for a sink, subscribe rights on the subscriptions for a source.
Connection fields
Set once, on the connection, in step 2 of the add-a-connection wizard.
| Field | Default | Description | |
|---|---|---|---|
project_id | Required | — | The Google Cloud project that owns the topics and subscriptions. |
credentials_file | Advanced | — | A service-account key file for the runner to use. Most connections attach the key in step 3 instead. |
enable_message_ordering | Advanced | false | Publish with ordering keys so subscribers receive messages for the same key in order. |
ordering_key | Advanced | — | The ordering key to publish with when ordering is on. |
max_retries | Advanced | 5 | How many times a failed publish is retried. |
max_outstanding_messages | Advanced | 1000 | How many messages may be in flight at once. |
Credentials
Step 3 asks for Credentials JSON: a service-account key, stored as a Blunox-managed secret or as a reference to your own secret store. Leave it empty when the runner already has an identity in the project. See Connections.