Apache Kafka

Apache Kafka can sit on either side of a sync. As a sink, Data Sync publishes each change as a message to a topic. As a source, it reads messages from topics. As a sink it is append-only, so delivery is at-least-once.

Source Sink append_only

Overview

Kafka is the usual way to fan database changes out to many consumers: replicate once into a topic, and let each downstream service read it at its own pace.

A topic is an append-only log. When a runner restarts and replays a window of changes from its last checkpoint, those changes are published again, so a consumer can see the same change twice. There is no key on the topic for Data Sync to converge on, which is why Kafka is append_only in the delivery guarantees matrix. Design consumers to be idempotent, so that applying the same change twice has no further effect.

Before you connect

Decide who creates topics. With Auto create topics on (the default), Data Sync creates the topics it needs. If your topics are managed by your own tooling, turn it off and create them first.
Open the network path from the runner to every broker in the list, not just the first one: clients talk to the broker that leads each partition.
Create credentials if your brokers use SASL, with permission to write to the target topics for a sink, or read from the source topics for a source.

Connection fields

Set once, on the connection, in step 2 of the add-a-connection wizard.

Field		Default	Description
`brokers`	Required	—	The bootstrap brokers, as `host:port`.
`client_id`	Optional	`pulse-sink`	The client ID Data Sync presents to the brokers. It appears in broker logs and is what client quotas match on.
`tls_enabled`	Optional	`false`	Connect to the brokers over TLS.
`auto_create_topics`	Optional	`true`	Let Data Sync create missing topics. Turn it off when topics are managed elsewhere.
`max_retries`	Advanced	`5`	How many times a failed request to the brokers is retried.

Credentials

Step 3 asks for a SASL mechanism, SASL username and SASL password. Leave all three empty for brokers that do not use SASL. Otherwise enter the mechanism your brokers are configured for, and store the secret as a Blunox-managed value or as a reference to your own secret store. See Connections.