Apache Iceberg
Apache Iceberg is a sink in Data Sync: changes from any source are written into Iceberg tables, registered in your catalog and stored in an S3-compatible bucket. There is no Iceberg source.
Overview
An Iceberg table has two halves: metadata in a catalog, and data files in object storage. The connection therefore has two halves too. The main fields point at the catalog. The storage fields, under Advanced settings, point at the bucket the data files are written to; the bucket is required even though it sits there.
Iceberg records deletes and updates as delete files alongside the data. Many of them slow reads down until the table is compacted, which is what the two delete-file warning thresholds watch for.
Before you connect
- Have a catalog Data Sync can reach, and note its URL. If it requires authentication, issue a token for Data Sync.
- Create the namespace the tables will live in.
- Create the bucket for the data files, and an access key with write access to it.
Connection fields
Set once, on the connection, in step 2 of the add-a-connection wizard.
| Field | Default | Description | |
|---|---|---|---|
catalog_url | Required | — | The catalog's URL. |
warehouse | Optional | — | The warehouse identifier, for catalogs that host more than one. |
namespace | Required | — | The namespace tables are created in. |
s3_bucket | Advanced · required | — | The bucket data files are written to. |
s3_prefix | Advanced | — | A path prefix inside the bucket. |
s3_endpoint | Advanced | Amazon S3 | The address of an S3-compatible service. Leave it empty for Amazon S3. |
s3_region | Advanced | us-east-1 | The bucket's region. |
s3_force_path_style | Advanced | false | Address the bucket as endpoint/bucket. Most S3-compatible services need this on. |
compression_codec | Advanced | snappy | Compression for data files: snappy, gzip, zstd or none. |
delete_file_warn_count | Advanced | 100 | Warn when a table has accumulated this many delete files. |
delete_file_warn_ratio | Advanced | 1.0 | Warn when a table's delete files reach this ratio to its data files. |
Credentials
Step 3 asks for a catalog token, and an S3 access key ID and S3 secret access key for the bucket. All three are optional in the form; supply whichever your catalog and bucket require. Store them as Blunox-managed secrets or as references to your own secret store. See Connections.