Since its launch in 2017, the Singer protocol has increasingly become the de facto open source standard for data movement, and Meltano believes Singer has the staying power to remain the de facto standard for years to come. Until now, however, there was no cross-functional governing body that could actively guide the development of the Singer standard and create best practices for the community. That changes today – as we officially announce that Meltano, along with seven other member companies, are joining forces to support the future growth of the Singer spec and the Singer ecosystem.
We believe open source thrives when people across different backgrounds and with different business models invest their best ideas and efforts. Read on if you agree!
A diverse coalition, coming together
As of the working group’s launch, our participants represent a diverse set of companies and interests joining in this effort from around the globe. We are brought together by the strong conviction that there’s more to be gained by working together than by working alone.
We have some fantastic companies on board at launch, but we are especially excited to have both Talend and Wise as founding member companies — who run StitchData.com and Pipelinewise, respectively. Stitch (now owned by Talend) are the original creators of the Singer spec, and we are thrilled to have them onboard.
Wise (formerly Transferwise) has a much-loved Singer platform called Pipelinewise, which offers yaml-based configuration of Singer pipelines — and has also piloted FAST_SYNC, a proprietary batch-based replication method for Singer taps and targets. Based on the success of this batch-based replication method, we’re already working together on a new spec to bring the batch-based FAST_SYNC protocol to the wider Singer community!
In alphabetical order, our eight founding member companies are:
- AppliedLabs – maintainers of multiple taps and owners of singerly.co, a hosted Meltano and Airflow platform
- AutoIDM – tap/target dev and support, data engineering, and reverse ETL consultancy
- Bytecode – data consultancy, maintainers of over 30 taps on the Hub
- Hotglue – data integration platform which leverages Singer for data movement
- Meltano – creators of Meltano, MeltanoHub, and the SDK for Singer Taps and Targets
- Talend – maintainers of Stitch, original founders of Singer
- Wise – creators of Pipelinewise
- Y42 – no-code data platform, maintainers of multiple Singer taps and targets
Shared values bring us together
We had our inaugural meeting on September 28 and while we each have different priorities, we came away with a few core themes that drive us all.
Improving performance
Several of us are motivated to introduce batch processing capabilities and other improvements, which can drastically improve sync speeds for a number of scenarios.
Expanding capabilities
Many member companies want to see more advanced capabilities enumerated, documented, and officially adopted. For example, a longstanding “unofficial” capability has been the “activate version” message, which when used in a full table sync, can trigger soft or hard deletion of stale records in the target table. Through this group’s efforts, we hope to bring this capability into a state of being fully documented and officially supported.
Sharing and promoting best practices
In working with Singer over the past many years, each of us have learned through experience certain tips and tricks to make taps and targets work in a reliable and performant way. Through this working group, we hope to codify and more broadly share our collective best practices with each other, and with the wider community.
Working with transparency and openness in mind
We each have our own values but we also believe in fostering openness and transparency in our efforts, to be inclusive of the wider Singer community in our efforts. That is why we created a public GitHub issue board for all community proposals, and we are committed to posting regular updates on the progress and action items coming from the group. Stay tuned for more on this topic in future posts!
Moving forward, together
In forming this new Singer Working Group, we collectively demonstrate our commitment to the Singer community, each of us motivated to see the Singer spec evolve and grow. While we are still creating foundational governance procedures (more on that in a future post), we are excited today to announce this joint effort to continue improving Singer and to make it better than ever for everyone in our community who relies on Singer to reliably replicate data every single day.
We are committed to the success of the open source data community – won’t you join us?