The Problem Nobody Budgets for
Most data teams don’t set out to become ingestion specialists. It happens gradually.
You start with a handful of sources. A CRM, a billing system, a production database. Someone writes a few scripts or picks a tool, and the data lands in the warehouse. Job done.
Then the sources multiply. An API changes its pagination. A schema drifts overnight. A connector silently stops pulling a column that finance relies on for month-end. Suddenly your most experienced engineers are spending their mornings in logs, working out why yesterday’s load didn’t finish.
None of this shows up in a roadmap. It shows up as a slow drain on attention. And attention is the one resource a data team can’t hire its way out of.
Why I’m saying this as the founder of an ingestion company
There’s an obvious tension here. My business depends on ingestion. Surely I want you to care about it deeply?
I don’t. I want you to care about it the way you care about electricity in your office. You expect it to work. You notice when it doesn’t. You don’t hold weekly meetings about it.
Here’s what I’ve learned from working with teams across financial services, SaaS and marketing: the value of data is never created at the point of ingestion. It’s created when someone makes a better decision, ships a feature, or catches a problem before a customer does. Ingestion is the plumbing that makes that possible. Important, yes. But plumbing.
If your team is talking about pipelines more than they’re talking about outcomes, something has gone wrong. And if I’m honest, the ingestion industry has helped create that problem by making tooling that demands too much of your attention.
Where the time actually goes
When I sit down with a data team and ask where their week goes, the same patterns come up.
- Firefighting failed loads. Jobs that break at 3am and need a human to restart them.
- Chasing upstream changes. Source systems change without warning, and the pipeline is the first thing to notice.
- Maintaining custom code. The script someone wrote two years ago that only one person understands.
- Debating tools. Evaluating, migrating, re-evaluating. Months of effort that often ends up back where it started.
- Explaining gaps to the business. Every missing row erodes trust, and rebuilding it takes far longer than losing it.
Each of these feels urgent. None of them moves the business forward.
What “minimal attention” actually looks like
Reducing time spent on ingestion isn’t about caring less. It’s about setting things up so you don’t have to care as often. In practice that means a few things.
Reliability is the default, not a project. Retries, alerting and recovery should be built in. If a human has to intervene for routine failures, the system isn’t finished.
Changes are handled, not discovered. Schema changes and API updates should be caught and managed automatically, or at least surfaced clearly before they break anything downstream.
Standard sources stay standard. There’s no competitive advantage in how you pull data from a popular CRM. Use proven connectors and save custom engineering for the sources that genuinely are unique to your business.
Ownership is clear. Someone should know who’s responsible for a pipeline, but that responsibility should take hours a month, not hours a day.
The tool decision gets made once. Pick something solid, commit to it, and stop relitigating. The cost of constant tool evaluation is almost always higher than the cost of a slightly imperfect choice.
A quick test you can run this week
Ask your data team one question: “How many hours last week went on keeping data flowing, rather than using it?”
Don’t aim for precision. A rough answer is enough.
If the number is more than 10 to 15 percent of the team’s time, ingestion is taking more than its share. If the answer is “we don’t really know,” that tells you something too, because problems that aren’t measured tend to grow quietly.
Then pick the single most time-consuming pipeline and ask whether it needs to be that way. Often it’s one fragile source, or one piece of custom code, that accounts for most of the pain.
The longer-term shift
Over the next six to twelve months, the goal is to move your team’s attention up the stack. Less time on moving data, more time on modelling it, questioning it and putting it in front of the people who make decisions.
This matters even more now. As teams start building AI into their products and workflows, the demand for clean, timely, trustworthy data is going up sharply. The teams that win won’t be the ones with the cleverest pipelines. They’ll be the ones who stopped thinking about pipelines early and put their best people on the problems that actually differentiate the business.
It’s a theme that comes up again and again when I talk to practitioners on LinkedIn Live and the Data Matas podcast. The most effective data leaders are almost boring about infrastructure. They made it dependable, and then they moved on.
What I want for our customers
If I’ve done my job well at Meltano, our customers shouldn’t think about us very often. They should notice that data arrives, that problems get flagged before they become incidents, and that their engineers have time for the work they were hired to do.
That’s a slightly odd ambition for a founder: to build something people forget about. But I think it’s the right one. The best infrastructure earns trust by being quiet.
So here’s my suggestion. Measure how much attention ingestion is costing you. Fix the worst offender. Then give that time back to the work that matters.
If you want to compare notes on where your team’s time is going, I’m always happy to have that conversation.
Frequently asked questions
Is managing Fivetran with Terraform the same as having pipelines in Git?
Not quite. Fivetran’s Terraform provider lets you manage connections as code, but it’s a separate layer with its own state, and the UI can still change things underneath it. In Meltano, the Git repo is the source of truth. There’s no second place for changes to happen.
Am I locked into Fivetran?
Less than it feels. Your data sits in your own warehouse, and your transformation logic sits in your dbt project. What Fivetran owns is the connectors and schedules, and those can be rebuilt and run in parallel before you switch anything off.
Do I have to move all my Fivetran connectors at once?
No. Most teams move connector by connector, starting with the highest-volume or most expensive sources. Mirror Mode means each one can be validated before it goes live.
What happens to my data in Fivetran after I switch?
Nothing. Data Fivetran loaded stays in your warehouse. You can keep those tables, archive them, or drop them once you’re confident in the new pipelines.
Can I keep using dbt if I move off Fivetran?
Yes. Meltano works with your existing dbt project and can orchestrate it for you. If you use Fivetran’s dbt packages, they’ll need a shim layer or a rewrite, because they expect Fivetran’s schema. You can also trigger existing SQL or stored procedures as pipeline steps if you’re not ready to move everything into dbt yet.
What’s the best Fivetran alternative for high-volume data?
For high-volume sources, pricing model matters more than anything else. Row-based pricing grows with every row you move. Meltano’s compute-based pricing grows with the work done, which is why the savings are largest where volumes are highest. See how the two compare on our Meltano vs Fivetran page.
How do I work out what I’d save?
Put your current volumes into the pricing calculator. It takes two minutes.
Find out what switching would save you
The easiest first step is seeing the number. Try our pricing calculator with your current Fivetran volumes.
If you’d rather talk it through, book a call and we’ll go through your connectors with you.
Ready to plan the move? Read How to migrate from Fivetran to Meltano for the step-by-step process.