People will find a way to ask
Navo described a useful motivation for building incident.io’s internal data brain. People were already using agents to explore business questions. The data team needed a way to make that activity more consistent and governable.
He was careful about what had actually happened. They hadn’t reached the point of competing answers causing a major dispute. He was getting requests to sense-check analyses whose workings weren’t clear to him.
That’s a familiar position for a data team. Someone brings you a conclusion and asks whether it’s right. You first have to work out what they asked, what the analysis used and which assumptions went into it.
If your response to conversational analytics is to wait until everything is ready, you may find the business has already started. A queue for dashboard changes is a strong incentive to try another route.
I’d rather give that curiosity somewhere useful to go, with a way to inspect how an answer was reached.
A metric definition only gets you so far
Agreeing how to calculate revenue matters. It doesn’t tell an agent everything it needs to know about a renewal conversation.
Navo described organising context around business themes, including renewals and retention. That brings together metric definitions with the things an analyst would explain to a colleague: which tables to use, what assumptions are reasonable, which signals matter and where the data needs treating carefully.
A leadership question about retention and a customer success question about an account at risk may draw on related data. They need different analysis.
Think about onboarding a new analyst. You wouldn’t hand them a schema and assume they now understood the business. You’d explain how customers behave, what the teams are trying to achieve and which apparently obvious conclusions have caught people out before.
Your agent needs access to that knowledge too.
The semantic layer still has a job
The practical question is where to begin. I’d choose one recurring business decision and write down what a capable analyst needs to know to support it. That gives you something specific to build, use and improve.
Oliver made a distinction that deserves more attention. A semantic layer can hold shared meaning. It can also provide a stable way to request data while the underlying implementation changes.
Those are useful properties even when an agent can write the SQL itself.
If someone builds an analysis directly against a table, they’ve created a dependency on that table. Rename a column or change the model and their work may break. When agents make it easier for people to build their own tools and reports, there can be more of those dependencies to maintain.
Oliver described the value of a contract between the people maintaining the data and the people consuming it. You can change how a metric is produced while preserving the agreed way to access it.
A semantic layer is one way to provide that contract. The important decision is what you want people and agents to be able to rely on, and who takes responsibility when it changes.
Generating SQL doesn’t remove that responsibility.
Test whether it knows when to stop
The most useful part of the discussion, for me, was Navo’s approach to evaluation. He described three types of question:
• Questions with an agreed answer. The system should return the right value consistently.
• Questions requiring judgement. There may be several reasonable answers, so assess the method and assumptions.
• Questions the available evidence cannot answer. The system should recognise the limit rather than produce a confident guess.
That last category deserves a place in every team’s tests.
A system that always returns an answer can look impressive in a demo. In use, I want to know whether it can tell someone that the evidence isn’t there.
Use questions your colleagues actually ask. Include awkward ones. If you only test neatly defined metrics, you’ll learn how well the system handles neatly defined metrics. You won’t learn much about the exploratory questions people will bring it on Monday morning.
Oliver also recommended reading the traces: following the steps an agent took to reach its answer. It takes time, but it shows you where the reasoning went wrong and which information was missing.
When you add context to fix a failure, test that case again. Navo’s point was that the new information should earn its place. More documentation isn’t automatically better guidance.
Your context needs reliable data beneath it
From our side at Meltano, I keep coming back to what happens before the question reaches the agent.
You can describe a metric precisely and still calculate it from an incomplete load. You can give an agent excellent business context and leave it working with yesterday’s data when the decision needs today’s.
So the readiness check has to include the pipelines. Which sources feed the answer? When did they last load successfully? How fresh does the data need to be for this particular decision? Who owns a failure?
Those questions become harder to ignore when answers can spread through the business faster than the data team can review them.
Give the data team better work
Navo described spending less time answering individual requests and more time building the system through which colleagues could answer their own questions. That also gave him room for more complex strategic work requiring a deeper understanding of the business.
That’s a change worth pursuing but someone still has to establish the definitions, understand the decisions and challenge the reasoning.
My advice is to start with a business question people care about, give the agent the context an analyst would need and inspect what it does. Include a question it should refuse to answer. Check the data feeding it. Then improve the parts that fail.
If you’re responsible for the data platform, you should be able to explain why an answer deserves to be trusted. A query that ran successfully is the beginning of that explanation.
Give your AI answers a foundation you can trust.
If unreliable pipelines or stale data are holding you back, talk to Meltano about getting your data where it needs to be, with pipelines you own and code you control.
Talk to us about your data stack →
Watch the full conversation with Navo Das and Oliver Laslett
What Happens When Everyone Can Talk to Their Data? https://youtu.be/_8fV39upsMU?si=L5WKf4Sm2WDjAybM