In a recent post, Douwe Maan (Meltano CEO), highlighted our mission and vision for Meltano. Part of that mission is our embrace of a better way of working using DataOps. At its simplest, DataOps is the application of software engineering best practices to data tooling.
One of the key components of DevOps is having isolated environments for building, testing, and deploying software. We believe data tooling should natively support this capability.
To that end, we’re excited to launch Meltano Environments!
Control Your Environment
As of Meltano v1.86.0, you can now quickly toggle the execution of commands between different configuration sets by passing a single `MELTANO_ENVIRONMENT` environment variable or the `–environment` CLI option:
meltano –environment=dev elt tap-github target-sqlite
Simply add a top level `environments` key to your `meltano.yml` file and you’ll bring the power of DataOps to your project:
# meltano.yaml
environments:
- name: dev
config:
plugins:
extractors:..
- name: prod
config:
plugins:
extractors:..Why is this a big deal?
With this command, you’re now able to avoid the hassle of toggling between multiple `.env` files and also reduce the number of variables that need to be set manually. Configuration set for plugins in an environment will inherit from the base plugin definition meaning you can add to or override any settings needed for specific deployment scenarios.
There’s now less for you as an analyst or data engineer to think about and manage, which gives you more confidence that your jobs will run predictably, every time. Creating isolated environments has never been easier: just give your new environment a name (dev, test, prod, etc.), configure it once, and you’re set. Now there’s no limit on the number of environments you can seamlessly manage with Meltano.
This is another step for Meltano towards our goals of bringing DataOps to teams everywhere.
Next Steps
In the spirit of Iteration, we’ve launched this knowing there are more capabilities we want to add. This feature is not yet available in the UI and we’re still iterating on the full command line interface for managing all of your environments.
We’re also using this new feature internally to add data from Google Analytics to MeltanoHub for individual connectors. With the environments feature, we’re now able to have a production environment, as well as CI and local dev environments, for pulling data from Google Analytics and other sources.
We’d love your feedback on this new feature and to better understand how you’re using it! Join us in Slack or come to our weekly Office Hours to share how it’s going, or you can file an issue if you find a bug. Happy building!