Today we’re excited to launch MeltanoLabs into the world. MeltanoLabs (not to be confused with dbt Labs!) is the new home for any Singer connector or Meltano plugin (extractors, loaders, transforms, etc.) that doesn’t otherwise have a place to live.
When we took a look at some of the challenges that existed in the Singer ecosystem, one that we identified was the opportunity to better support a community of users and contributors to better maintain connectors.
Connector Ownership Models
AJ, Head of Engineering at Meltano, laid out 4 general models of ownership for any given connector. The ownership models are as follows:
The first is the “Single Named Owner” model. The `singer-io` GitHub namespace fits into this category. One company manages all of the connectors in the namespace but eventually reaches a scaling limit. This can work for a lot of organizations, consultancies included, but there is limited incentive to add features the organizations themselves don’t need.
The second is the “Vendor Self-Managed” model. This is where an organization maintains its own tap or target in its own namespace. A great example of this is the Decentraland tap. We like this model and would love to see more companies follow this path. We aim to encourage more organizations to embrace this model.
The third is the “Benevolent Community Member” model. This is where an individual has forked a tap, adding some features, and now is seen as the primary maintainer of the connector. This model is great in the short-term, but it has its downsides as individuals may not be able to always maintain a connector.
Each of these have their own pros and cons and we think they’re all worthwhile paths for connector maintenance. However, we believe there’s room for another model that addresses many of the concerns with the above approaches.
With MeltanoLabs we’re building the 4th ownership model we call “Community-Managed Fork with Community Maintainers”.
MeltanoLabs on GitHub
We see MeltanoLabs as a combination of the first and third ownership models. A single company, Meltano in this case, will manage the GitHub org, but we’re being explicit that we as a company will not be maintaining any of the plugins that may live there. We want to bring community members on as members of the GitHub org who are willing and able to help provide plugin maintenance.
We’ll be leveraging GitHub’s Teams feature to enable project access for different plugins. For users or organizations that may not want to host a connector in their own namespace, we can host it in MeltanoLabs. The next step would be to add specific people as maintainers who will continue to develop the connector and provide some support such as issue and PR triaging and review.
For Singer connectors in particular, one big benefit we see to this model is the graceful transition of a connector to different maintainers. As individuals change organizations their ability to maintain a connector may change as well. Within MeltanoLabs we want to support the adoption and transfer of connector maintainership so that the Singer and wider data communities can still rely on the existing codebase.
As with all things we do, we’re going to be iterating on this model over time as we work with the community.
What this means in the short-term
For Meltano, this means that we’re going to deprecate any connector that lived in the Meltano GitLab group. We will be transitioning the discoverable variants to community supported ones over the next few weeks.
Development of Meltano, the Meltano SDK, and MeltanoHub will still take place on GitLab and any issue related to those projects should be opened there. The Singer Working Group and other community efforts will live on GitHub in the MeltanoLabs group.
As part of the acceptance process for MeltanoLabs we will be requiring some level of commitment and a minimum level of quality. Any connector written on the Meltano SDK will meet this bar, but that will not be a hard requirement for acceptance. The exact details of this, as well as the security and permissions processes we implement, will be iterated on in collaboration with the community over the coming weeks and months.
A Bright Future for Singer and Meltano
If this model sounds interesting to you, please reach out to us via Slack. There we can work with you to move your connector over to MeltanoLabs.
We’re excited by this effort and hope it can provide more stability and a higher level of quality for the whole community. If you have any feedback, please let us know by joining Slack or opening an issue.