MySQL and MariaDB are popular open-source relational database management systems.

MySQL and MariaDB are powerful and flexible database management systems that allow users to store, organize, and retrieve data efficiently. They are widely used in web applications and are known for their reliability, scalability, and ease of use. MySQL and MariaDB support a wide range of programming languages and platforms, making them a popular choice for developers and businesses of all sizes. They offer features such as transaction support, replication, and clustering, and can be used for a variety of applications, from small personal projects to large enterprise systems. Overall, MySQL/MariaDB are versatile and reliable database management systems that are widely used in the industry.

Setup

1. Create a database user

Create a dedicated user for the tap and grant the minimum permissions required for your chosen replication method.

FULL_TABLE and INCREMENTAL replication — read access to the target schema(s) is sufficient:

CREATE USER 'tap_mysql'@'%' IDENTIFIED BY '<password>';
GRANT SELECT ON <database>.* TO 'tap_mysql'@'%';
FLUSH PRIVILEGES;

LOG_BASED replication (binlog streaming) — the user additionally needs replication privileges granted at the global level:

CREATE USER 'tap_mysql'@'%' IDENTIFIED BY '<password>';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'tap_mysql'@'%';
FLUSH PRIVILEGES;

2. Enable binary logging (LOG_BASED replication only)

Binary logging must be active and set to ROW format. Add the following to your MySQL/MariaDB server configuration (my.cnf / my.ini) and restart the server:

[mysqld]
log_bin            = mysql-bin
binlog_format      = ROW
expire_logs_days   = 7   # retain enough history to survive pipeline gaps

For managed services, enable binary logging via the provider console:

• Amazon RDS / Aurora — set the binlog_format parameter to ROW in your parameter group and enable automated backups (required for binlog).
• Google Cloud SQL — enable "Point-in-time recovery" and set binlog_format = ROW in the database flags.
• Azure Database for MySQL — enable "Backup retention" and set binlog_row_image = FULL in server parameters.

3. Co-locate your pipeline for optimal backfill performance

During an initial full-table backfill the tap reads every row from MySQL over the network. Running the pipeline in the same cloud region as your MySQL server eliminates cross-region egress latency and bandwidth costs, which can reduce backfill time by an order of magnitude for large tables.

• Amazon RDS/Aurora — deploy the pipeline in the same AWS region (e.g. eu-west-1).
• Google Cloud SQL — deploy in the same GCP region (e.g. europe-west2).
• Azure Database for MySQL — deploy in the same Azure region (e.g. uksouth).
• Self-hosted MySQL — if the database is on-premises or in a private network, use the SSH tunnel settings (ssh_tunnel.*) to reach it without exposing the port publicly.

4. Enable BATCH mode for large-table performance (optional)

Instead of emitting one record per row, the tap can write compressed JSONL files and emit batch messages, which compatible targets can load in bulk. For large tables this typically reduces pipeline wall-clock time significantly because the target can load pre-aggregated compressed files in a single bulk operation rather than processing individual records one at a time.

To enable BATCH mode, set batch_size_rows to a positive integer (e.g. 200000). The tap will then emit batch messages instead of individual records.

Target compatibility caveat — not all targets support batch messages. Only targets that advertise the batch capability will process them correctly. If your target does not support it, leave batch_size_rows unset or at 0 to stay in record-per-row mode.

Settings

Host

The hostname or IP address of the MySQL/MariaDB server.

Port

The port number to use for the connection.

User

The username to use for authentication.

Password

The password to use for authentication.

Database

The name of the database to connect to.

Cursor Class

The class to use for the cursor.

Server ID

The server ID to use for replication.

Filter DBs

A list of databases to include or exclude from replication.

Use GTID

Whether to use global transaction identifiers for replication.

Engine

The storage engine to use for the connection.

SSL

Whether to use SSL encryption for the connection.

SSL CA

The path to the SSL CA certificate file.

SSL Certificate

The path to the SSL certificate file.

SSL Key

The path to the SSL key file.

Internal Hostname

The internal hostname to use for the connection.

Session SQLs

A list of SQL statements to execute when the connection is established.

SSH Tunnel Host

Address of the bastion host, this is the host we'll connect to via ssh

SSH Tunnel Port

Port to connect to bastion host

SSH Tunnel Private Key

A base64 encoded Private Key for authentication to the bastion host w/ key pair auth

SSH Tunnel Private Key Password

Private Key Password, leave None if no password is set

SSH Tunnel Username

Username to connect to bastion host

SSH Tunnel Password

Password to connect to bastion host w/ basic auth

Batch Format

The format of the batch files. Only 'arrow' and 'jsonl' are supported. Defaults to 'jsonl' if not specified.

Batch Size

Number of rows per BATCH message file.