Databricks is a unified data and AI platform built on a lakehouse architecture. This loader writes records into Delta tables in Unity Catalog through a Databricks SQL warehouse, using INSERT or MERGE statements depending on the load method.

Prerequisites

1. Server Hostname and HTTP Path: open your SQL warehouse in the Databricks workspace and copy both values from the Connection details tab.

2. Credentials: authenticate with one of the following.

• Personal access token (auth_type: pat, the default) — generate a token under Settings > Developer > Access tokens and set it as access_token.
• Service principal (auth_type: oauth_m2m) — create an OAuth secret for a service principal and set client_id and client_secret. Recommended for unattended use.

1. Permissions: the user or service principal needs CAN USE on the SQL warehouse, USE CATALOG on the catalog, and USE SCHEMA, CREATE SCHEMA and CREATE TABLE (or equivalent ownership) on the target schema. Large batches are staged as Parquet in a Unity Catalog volume named meltano_staging, created in the target schema if missing, so the principal also needs CREATE VOLUME and WRITE VOLUME. Without them the loader logs a warning once and falls back to slower inline inserts.

Target schema

Set default_target_schema to load every stream into one schema. If it is unset, the schema is taken from <schema>-<table> stream names. The schema is created if it does not exist; use schema_creation_parameters.retention_days to configure how long dropped tables remain recoverable with UNDROP TABLE in schemas created this way.

Settings

Server Hostname

Workspace hostname, e.g. dbc-1234.cloud.databricks.com.

HTTP Path

SQL warehouse HTTP path, e.g. /sql/1.0/warehouses/abc.

Authentication Type

pat for a personal access token, oauth_m2m for a service principal (client ID and secret).

Access Token

Personal access token. Required when auth_type is pat.

Client ID

Service principal client ID. Required for oauth_m2m.

Client Secret

Service principal OAuth secret. Required for oauth_m2m.

Catalog

Unity Catalog name. Defaults to the warehouse default.

Default Target Schema

Schema to load into. If unset, it is taken from <schema>-<table> stream names.

Load Method

upsert merges on the stream key properties, append-only always inserts, overwrite truncates each table once per run then loads.

Hard Delete

On ACTIVATE_VERSION, delete stale rows instead of marking them with _sdc_deleted_at.

Schema Retention Days

Optionally sets the recovery period for dropped managed tables in the schema, the period during which dropped tables can be recovered using the UNDROP TABLE command. If not specified, the schema inherits the recovery period from its parent catalog (default 7 days). Set the value to 0 to disable recovery, or between 7-30 days, inclusive.

Add Record Metadata

Whether to add metadata columns.

Batch Size Rows

Maximum number of rows in each batch. Each batch is staged and loaded with a fixed cost of a few seconds, so larger batches load faster. Lower it if your records are very wide.

Clean Up Batch Files

Delete Arrow BATCH files once they are loaded. These files are consume-once, so this is safe to leave enabled.

Validate Records

Whether to validate the schema of the incoming streams.

Flattening Enabled

'True' to enable schema flattening and automatically expand nested properties.

Flattening Max Depth

The max depth to flatten schemas.

Stream Map Config

User-defined config values to be used within map expressions.

Stream Maps

Config object for stream maps capability. For more information check out Stream Maps.