The Schema Nobody Documented
Every data team has one. The column that means something different than its name suggests. The table that was deprecated two years ago but still gets joined. The metric definition that lives in one person's memory and nowhere else. These gaps are not documentation problems. They are the reason AI query tools fail.
The Schema Nobody Documented
A company buys an AI query tool. The demo was impressive. The sales team promised natural language access to data, no SQL required, answers in seconds.
The tool gets connected to the database. A user types a question: how many active customers do we have?
The tool returns a number. It is wrong. Not slightly wrong. Off by a factor of three.
The investigation takes two days. The column called status does not mean what everyone assumed. In the application code, status = 'active' includes trial accounts that have never logged in. In the billing system, active means a paid subscription. In the CRM, active means any account with a logged interaction in the last ninety days. Three definitions, one column name, and no documentation anywhere that says which one the database actually contains.
The tool did exactly what it was asked. It read the column name, inferred a meaning, and wrote a query. The failure was not in the query. It was in the gap between what the schema said and what the business meant.
The column names are there. The meanings are somewhere else.
The Problem Is Not the Model
When an AI query tool gives a wrong answer, the instinct is to blame the model. A better model will fix it. This instinct is understandable, and it is almost always wrong.
A language model receives a question and whatever context the system provides. If that context is a list of table names and column names, the model has to infer meaning from naming conventions. It guesses that revenue includes tax or excludes it. It guesses that churn_date is the date the customer asked to leave or the date access was revoked. It guesses that a null in closed_at means the ticket is still open or that it was never tracked.
Every one of those guesses is reasonable. Most of them will be wrong for this specific business, because the correct answers live in business rules that were never written into the schema.
A data engineer on Reddit described the pattern precisely: "Even with detailed schemas, event taxonomies, metric definitions, it still often got the SQL logic wrong from a business perspective. The code was error free but it used the wrong columns or filtered the data incorrectly and thus the result was incorrect." https://www.reddit.com/r/dataengineering/comments/1k862la/would_you_use_this_tool_ai_that_writes_sql/
The model did not fail. The system around it failed to provide the one thing that would have made the answer correct: a documented, verified understanding of what the data actually means.
A model without documentation is doing pattern matching on column names. It will produce queries that run. It will not produce queries that are right.
The answer looked correct. That was the problem.
Silent Failure Is Worse Than a Crash
A wrong query that crashes is a bug. It returns an error, blocks the pipeline, forces someone to look. A wrong query that succeeds and returns a plausible number is something else entirely.
A practitioner testing text-to-SQL documented the pattern: on 32 questions, the system produced 7 silently wrong answers. The queries ran. The numbers looked normal. Nothing in the output indicated a problem. https://www.linkedin.com/posts/nikodemdabski_silent-failures-in-text-to-sql-a-growing-concern-activity-7455981620712341504-Qf8C
Another analysis found that 38.5% of analyst questions returned no records when the data was right there, a failure mode that logs nothing and alerts nothing. https://dev.to/ (the article is titled "38% of an analyst's questions get 'no records found' when the data is right there")
A commenter on Reddit's r/dataengineering put it plainly: "it's very difficult to validate the output of the results." The same thread included a response from someone who had built a prototype with detailed schemas and metric definitions and still found that the system "often got the SQL logic wrong from a business perspective." https://www.reddit.com/r/dataengineering/comments/1k862la/would_you_use_this_tool_ai_that_writes_sql/
The failure mode is not the query. It is the silence around the query. A number that arrives without provenance, without definition, and without a check against a trusted source is not an answer. It is a guess that looks like an answer, and it will be treated as fact until someone discovers otherwise.
What the Community Is Actually Saying
This is not a theoretical concern raised by skeptics. It is the consistent experience of people who have tried to deploy text-to-SQL in production.
On r/dataengineering, a commenter wrote: "I do not believe 'no matter how complex the query is, the platform generates the correct SQL.' It never does." https://www.reddit.com/r/dataengineering/comments/1k8632s/would_you_use_this_tool_ai_that_writes_sql/
A LinkedIn post about Databricks Genie described a test where the system was asked for a metric on a date where no row existed. Genie returned a specific number with a full reasoning trail, having silently assumed a value and hardcoded it into the query. The author's conclusion: "It never said 'I don't know.' That's how text-to-SQL fails, and without a semantic layer holding the definitions, I don't see how Agent Mode escapes it." https://www.linkedin.com/posts/nivbelleli_databricks-new-genie-agent-mode-did-everything-activity-7455503784778227713-lJav
On the Google Developer forums, a user reported that the conversational analytics engine "keeps on hallucinating about tables and columns despite them being clearly defined in the data-sources." https://discuss.google.dev/t/conversational-analytics-broken-sdk/325633
The pattern repeats across products, platforms, and communities. The problem is not that models cannot write SQL. It is that they are being asked to write SQL against a schema that has never been properly explained to them.
The same complaint, from different people, about different tools, for the same reason.
The Missing Layer
If the problem is missing documentation, the solution follows directly: the system needs a documented understanding of the schema that is verified by the people who know the business.
This is not a new idea. Data catalogs and semantic layers have existed for years. But most AI query tools skip this step, because it is slow, manual, and does not demo well. They connect to the database, read the column names, and start generating queries. The results are inconsistent, and the inconsistency is blamed on the model.
A better approach is to treat schema documentation as a first-class part of the system, not an optional add-on. The tool should propose a documentation of the schema, and the customer should review and approve it. Every field should be clarified by someone who knows what it means. Every relationship should be confirmed. Every ambiguous column should be resolved before a single query is generated.
This is the approach BlunxAI takes. The platform proposes automatic documentation of the schema, and the user approves it. The documentation becomes the reference against which every generated query is validated. The multi-agent pipeline includes a Validator step that checks results against the documented schema before they are returned. A query that joins on a relationship the schema does not define is rejected. A column that has not been clarified is flagged. https://blunxai.com/features
The difference is not that the model is better. The difference is that the model is working against a documented model of the business, not a list of column names.
The fix is not a larger model. The fix is a verified definition of what the data means, applied before generation.
The documentation is the guardrail. Without it, the query is a guess.
How Validation Actually Works
Documentation alone is not enough. The system also needs to check its work.
A multi-agent pipeline is one way to structure that check. Instead of a single model generating a query and returning it, the work is split into stages. One agent analyzes the question and identifies the relevant data. Another plans the collection strategy. A third collects the data. A fourth validates the result against the documented schema. A fifth synthesizes the answer into a chart, a table, and a summary. https://blunxai.com/features
The validation step is the one that matters most for correctness. It is the point where the system asks: does this result make sense given what we know about the schema? If the query joins on a relationship that has not been documented, the validator can reject it. If the result is empty when the data should be present, the validator can flag it. If the number is wildly outside the expected range, the validator can surface a warning instead of returning the number as fact.
This is not a guarantee of correctness. No system can guarantee that. But it is a check against the most common failure mode: a query that runs, returns a number, and is wrong for reasons the model could not have known.
A LinkedIn post on silent failures in text-to-SQL made the point directly: "A system that's 70% accurate but knows when it's wrong is more useful than one that's 75% accurate and silently serves bad answers." https://www.linkedin.com/posts/nikodemdabski_silent-failures-in-text-to-sql-a-growing-concern-activity-7455981620712341504-Qf8C
The validator is the mechanism that lets the system know when it is wrong. It is not glamorous, and it is the thing that makes the difference between a tool you can trust and a tool you have to double-check.
The check is the product. Everything else is generation.
The Honest Position
AI query tools are not going away. The ability to ask a question in plain language and get an answer is too valuable to abandon because the first generation was unreliable.
But the industry has spent the last year learning an uncomfortable lesson: a capable model and a database connection is not a product. The interesting work is in the layer that sits between them, the layer that documents the schema, validates the query, and refuses to answer when the answer cannot be trusted.
Until that layer exists, every AI-generated number carries a hidden question: did the query run because it was right, or did it run because nothing stopped it.
The difference between those two outcomes is the difference between analytics you can build a decision on and analytics that will eventually, quietly, be wrong.
Want to see what schema documentation and query validation look like in practice, with a multi-agent pipeline that checks its work before returning a result? https://blunxai.com/features