Dagster is an open-source data orchestrator for machine learning, analytics, and ETL.

Dagster provides a unified framework for building data pipelines that allows developers to define the inputs, outputs, and dependencies of each step in the pipeline, making it easier to test, maintain, and scale complex data workflows. It also includes features such as data validation, error handling, and monitoring to ensure the reliability and quality of data processing. Dagster supports a variety of data sources and execution environments, including local development, cloud-based services, and containerized deployments.

Settings

Auth Token

A token used for authentication when connecting to the Dagster API.

Start Date

The date from which to start streaming data.

Api Url

The URL of the Dagster API.

Stream Maps

A list of stream maps to use when streaming data.

Stream Map Config

Configuration settings for the stream maps.

Flattening Enabled

Whether or not to flatten the data when streaming.

Flattening Max Depth

The maximum depth to which the data should be flattened.