Elasticsearch

Elasticsearch is a source only in Data Sync — there is no Elasticsearch sink. It is also the one source in these docs that is not log-based, and that difference changes what you can build on it. Read the limitation below before you design around this connector.

Overview

Supported versions are 7.10 and later, 8.x and 9.x. The floor is 7.10 because the snapshot phase uses point-in-time search, which does not exist before that release.

Every other source in these docs reads a change log — the PostgreSQL write-ahead log, SQL Server's CDC tables, a Spanner change stream. Elasticsearch has no such log to tail. Data Sync therefore reads it by polling: it asks repeatedly for documents whose watermark field has moved past the last value it saw. That works well for append-and-update workloads, and it has one consequence that is not recoverable by tuning.

How it reads

A sync in the default snapshot_then_cdc mode runs two phases, and they read the index in genuinely different ways.

Snapshot. Data Sync opens a point-in-time reader, which pins a consistent view of the index for the duration of the backfill, and paginates it with sliced search_after. By default it uses one slice per primary shard, which is the natural parallelism boundary — each slice reads shard-local data, so slices do not contend. Set snapshot_slices if you want to override that.

CDC. Once the backfill is done, Data Sync switches to polling on the watermark field. Each round asks for documents whose watermark is:

• greater than the last watermark value it processed, and
• older than now − cdc_safety_lag.

Results are sorted by the watermark and paginated with search_after, and the highest watermark seen becomes the lower bound for the next round. Between rounds the connector waits cdc_poll_interval.

The upper bound is the part people delete first and regret later. It has its own section.

Choosing a watermark field

This is the most important configuration decision on this connector. The watermark is the only thing telling Data Sync which documents are new, so a field that does not move forward on every write means changes are silently never read.

The field must be two things:

• Monotonically increasing — it must be updated to a larger value every time the document changes. A created_at that is written once and never touched again will replicate the initial write and nothing after it.
• Sortable — the CDC query sorts on it and pages with search_after, both of which need a field type that supports sorting.

A date field is the natural choice, and the default cdc_watermark_field is @timestamp for exactly that reason. If your indexing pipeline stamps an update time on every write, use that.

A bare text field is not sortable — it is analysed into terms, and there is no single value to order by. Where a text field has a keyword sub-field, Data Sync uses the .keyword form automatically, so you can name the parent field and it will resolve. Where it does not, the field cannot be used.

The cdc_safety_lag window

cdc_safety_lag (default 30s) is the window of recent time that Data Sync refuses to read into. Each poll's upper bound is now − cdc_safety_lag, so the most recent half-minute of documents is deliberately left alone until the next round.

The window exists because a watermark is not a log position. Two writes stamped a millisecond apart can become visible to search in the opposite order, a document can be indexed with a slightly older timestamp than one already read, and a page boundary can fall between two documents that share a watermark value. Reading right up to now would step over all three cases and lose the documents involved — permanently, because the lower bound has already moved past them.

Lowering cdc_safety_lag reduces end-to-end latency by exactly the amount you remove. It also shrinks the margin absorbing out-of-order and late-arriving writes, so set it low only if you know your indexing path is tight, and treat missing near-boundary documents as the failure mode to watch for.

Connection fields

Set once, on the Connection, in step 2 of the add-a-connection wizard.

Fill in hosts or a cloud ID, not both. Credentials are collected in step 3 and stored as a reference, so the key itself never lives in the connection record.

Sync fields

Set per sync. These control which indices are read and how the two phases are paced.

How indices appear

Data Sync's mapping layer is built around schemas and tables, and Elasticsearch has neither. The connector bridges that with a synthetic schema named indices, so an index called orders is addressed as the table indices.orders everywhere in the wizard and in the mappings.

Each document's _id is the primary key. That is what makes replays converge at a keyed sink: the same document re-read after a restart upserts onto itself rather than duplicating.

The Arrow schema prepends two synthetic string columns before the document's own fields:

Metadata fields other than those two are skipped. Document fields follow, flattened to dotted-path columns.

Type mapping

Elasticsearch mapping type on the left, the Arrow type Data Sync produces on the right. Each sink then maps Arrow to its own types — see the sink connector's page for that half.

Limitations

Collected in one place, because several of them follow from the polling model rather than from anything you configured.