MongoDB is a NoSQL database program.

MongoDB is a document-oriented database program that uses JSON-like documents with optional schemas, providing a flexible and scalable solution for managing and storing data. It allows for easy integration with various programming languages and frameworks, and offers features such as automatic sharding, replication, and indexing for efficient data management.

Streams

Streams are dynamically discovered from the connected database at runtime. The available streams depend on the collections the configured user has access to.

Prerequisites

To obtain the required settings for connecting to MongoDB, you can follow these steps:

1. Connection String: MongoDB uses a connection string URI format for connections. You can obtain this from your MongoDB Atlas dashboard, your hosting provider, or your system administrator. For local deployments, it's typically mongodb://localhost:27017. For CDC (LOG_BASED replication), the connection string must include the replica set parameter, e.g., mongodb://localhost:27017/?replicaSet=rs0.

2. Database: The database name is the name of the MongoDB database that you want to connect to. You can obtain the database name from your system administrator or the hosting provider.

3. Authentication: If your MongoDB instance requires authentication, you'll need a username and password. These should be included in the connection string or obtained from your system administrator.

4. For Change Data Capture (CDC): If you want to use LOG_BASED replication mode, your MongoDB deployment must be running as a replica set or sharded cluster.

Setting Up Change Data Capture (CDC)

To use LOG_BASED replication mode (CDC), your MongoDB deployment must support change streams:

MongoDB Requirements:

• MongoDB 3.6+ running as a replica set or sharded cluster
• Connection string must include ?replicaSet=rs0

Local Development Setup:

# Start a 3-node replica set with Docker Compose
docker compose up -d

# Verify replica set is ready (wait 30-45 seconds)
docker exec mongodb-primary mongosh --eval "rs.status()" --quiet

Production: Ensure your MongoDB deployment is configured as a replica set (3+ nodes) or sharded cluster.

Connection String Example:

mongodb://username:password@host1:27017,host2:27017,host3:27017/?replicaSet=rs0

AWS DocumentDB Requirements:

• Cluster mode (not single instance)
• Change streams enabled per collection

Enable Change Streams:

db.adminCommand({
  modifyChangeStreams: 1,
  database: "your_database",
  collection: "your_collection",
  enable: true
})

Alternatively, set allow_modify_change_streams: true in the tap configuration to auto-enable (requires admin privileges, may incur additional costs).

Connection String Example:

mongodb://username:password@cluster.region.docdb.amazonaws.com:27017/?replicaSet=rs0&tls=true&tlsCAFile=rds-combined-ca-bundle.pem

Permissions: The MongoDB user must have:

• find permission on collections
• changeStream permission on collections
• Built-in roles like readAnyDatabase@admin or read@database_name provide these permissions

Settings

Database

Database name from which records will be extracted.

Mongodb Connection String

MongoDB connection string. See https://www.mongodb.com/docs/manual/reference/connection-string/#connection-string-uri-format for specification.

Documentdb Credential Json String

String (serialized JSON object) with keys 'username', 'password', 'engine', 'host', 'port', 'dbClusterIdentifier' or 'dbName', 'ssl'. See example at https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_secret_json_structure.html#reference_secret_json_structure_docdb. The password from this JSON object will be url-encoded by the tap before opening the database connection.

Documentdb Credential Json Extra Options

String (serialized JSON object) containing string-string key-value pairs which will be added to the connection string options when using documentdb_credential_json_string. For example, when set to the string {"tls":"true","tlsCAFile":"my-ca-bundle.pem"}, the options tls=true&tlsCAFile=my-ca-bundle.pem will be passed to the MongoClient.

Datetime Conversion

Parameter passed to MongoClient 'datetime_conversion' parameter. See documentation at https://pymongo.readthedocs.io/en/stable/examples/datetimes.html#handling-out-of-range-datetimes for details. The default value is 'datetime', which will throw a bson. Errors. InvalidBson error if a document contains a date outside the range of datetime. MINYEAR (year 1) to datetime. MAXYEAR (9999). The allowed values correspond to the enumeration members here: https://github.com/mongodb/mongo-python-driver/blob/e23eb7691e6e2905a77fc39a114d000ddf057e47/bson/codec_options.py#L192-L224 (they will be uppercased by the tap).

Prefix

An optional prefix which will be added to each stream name.

Filter Collections

Collections to discover (default: all) - filtering is case-insensitive. Useful for improving catalog discovery performance.

Start Date

Start date. This is used for incremental replication only. Log based replication does not support this setting - do not provide it unless using the incremental replication method. Defaults to epoch zero time 1970-01-01 if tap uses incremental replication method.

Add Record Metadata

When True, _sdc metadata fields will be added to records produced by this tap.

Allow Modify Change Streams

In DocumentDB (unlike MongoDB), change streams must be enabled specifically (see https://docs.aws.amazon.com/documentdb/latest/developerguide/change_streams.html#change_streams-enabling). If attempting to open a change stream against a collection on which change streams have not been enabled, an OperationFailure error will be raised. If this property is set to True, when this error is seen, the tap will execute an admin command to enable change streams and then retry the read operation. Note: this may incur new costs in AWS DocumentDB, and it requires elevated permissions onthe user - the user must have the modifyChangeStreams permission in addition to read permissions.

Operation Types

List of MongoDB change stream operation types to include in tap output. The default behavior is to limit to document-level operation types. See full list of operation types athttps://www.mongodb.com/docs/manual/reference/change-events/#operation-types. Note that the list of allowed_values for this property includes some values not available to all MongoDB versions.

Change Stream Pipeline

Optional list of MongoDB aggregation pipeline stages to append to the change stream, passed as the pipeline argument to collection.watch(). Only used in LOG_BASED replication mode. For example, [{"$unset": "updateDescription"}] drops the updateDescription field (which the tap does not read) from update events, helping avoid MongoDB's 16MB BSON document size limit on events for collections with large documents. No validation is performed on this pipeline: it must not remove fields the tap depends on (_id, operationType, clusterTime, ns, and fullDocument/documentKey).

Document Layout

How documents are emitted. object emits each document as a single document object. flat emits each document field declared in the stream's schema as its own column: declare the fields as properties of the stream's catalog schema (e.g. with Meltano's schema extra). Values of BSON types JSON Schema can't express (ObjectId, Decimal128, ...) in a string property are converted to strings, a value of any other type that doesn't match its declared type fails the sync, undeclared fields are dropped with a warning, and a missing field is null.

Stream Maps

Stream maps. See https://sdk.meltano.com/en/latest/stream_maps.html for documentation.

Stream Map Config

Stream map config. See https://sdk.meltano.com/en/latest/stream_maps.html for documentation.

Faker Seed

Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator

Faker Locale

One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization

Enable Schema Flattening

'True' to enable schema flattening and automatically expand nested properties.

Max Flattening Depth

The max depth to flatten schemas.

Max Key Length

The maximum length of a flattened key.

Batch Encoding Format

Format to use for batch files. With arrow, incremental streams write Apache Arrow IPC files decoded straight from BSON; LOG_BASED streams keep emitting RECORD messages.

Batch Compression Format

Compression format to use for batch files.

Batch Storage Root

Root path to use when writing batch files.

Batch Storage Prefix

Prefix to use when writing batch files.

Batch Size

Max number of rows per batch file. Defaults to 100000 with the arrow format.