Vizuators' blog
“ChatGPT can write SQL” — and other misleading demos
Why most AI tools fail on real Tableau dashboards — and how to actually see what they‘re doing
“ChatGPT can write SQL” is one of the most misleading lines in analytics right now.
Not because it‘s false. Because of what it leaves out.
Writing a query in a sandbox is easy. Putting AI on a dashboard that 200 people rely on — that‘s where it falls apart.
Not because it‘s false. Because of what it leaves out.
Writing a query in a sandbox is easy. Putting AI on a dashboard that 200 people rely on — that‘s where it falls apart.
The demo that started it all
Because here’s what happens when the AI is let loose on real data.
It invents fields that don‘t exist. It confidently references a column that was never there. It misreads calculated measures because it doesn’t understand the business logic behind them. And then it hands back a number — clean, formatted, completely confident — that is simply wrong.
And no one can tell. Because no one saw how it got there.
The demo never shows that part. Your dashboard will.
It invents fields that don‘t exist. It confidently references a column that was never there. It misreads calculated measures because it doesn’t understand the business logic behind them. And then it hands back a number — clean, formatted, completely confident — that is simply wrong.
And no one can tell. Because no one saw how it got there.
The demo never shows that part. Your dashboard will.
A real example
Let us show you what this looks like in practice.
We asked an assistant to write a query against a sales dashboard. It returned a long, complex SQL with multiple CTEs and window functions — the kind of code that looks impressive at first glance.
But at the very bottom, hidden among all that careful logic, it was grouping by a column that didn‘t even exist: ep.UserId.
The model had guessed that a column named UserId must be there. It felt plausible. But the actual column was named CreatedByUserId.
It looked confident. It looked correct. It was wrong.
This is the danger: the model doesn’t know when it doesn‘t know. It makes a guess and presents it as fact. And because the rest of the code looks so solid, you’d never think to question it.
We asked an assistant to write a query against a sales dashboard. It returned a long, complex SQL with multiple CTEs and window functions — the kind of code that looks impressive at first glance.
But at the very bottom, hidden among all that careful logic, it was grouping by a column that didn‘t even exist: ep.UserId.
The model had guessed that a column named UserId must be there. It felt plausible. But the actual column was named CreatedByUserId.
It looked confident. It looked correct. It was wrong.
This is the danger: the model doesn’t know when it doesn‘t know. It makes a guess and presents it as fact. And because the rest of the code looks so solid, you’d never think to question it.
Why this keeps happening
Language models are pattern-matchers, not database administrators. They don't actually understand your schema — they guess based on patterns they've seen in other code.
If a field name sounds familiar, they‘ll use it. They won’t stop to check whether it actually exists in your data source. They assume.
And the worst part? They won‘t tell you they’re guessing. They just hand you a confident answer and move on.
This is what we mean when we talk about hallucinations: not wild nonsense, but plausible-sounding wrongness. The kind that slips through because it looks right.
If a field name sounds familiar, they‘ll use it. They won’t stop to check whether it actually exists in your data source. They assume.
And the worst part? They won‘t tell you they’re guessing. They just hand you a confident answer and move on.
This is what we mean when we talk about hallucinations: not wild nonsense, but plausible-sounding wrongness. The kind that slips through because it looks right.
How Events mode changes this
That‘s why we built Events mode.
Instead of just showing you the final answer, it shows you everything that happened behind the scenes:
If something looks off, you can trace it back. Check the query. Check the data. Check the prompt. See exactly where the problem started.
Instead of just showing you the final answer, it shows you everything that happened behind the scenes:
- The exact query it sent to Tableau. You see whether it interpreted your question correctly.
- The data that came back. You check whether the numbers match what you expected.
- The prompt it sent to the LLM. You see how much context the model actually got.
- The model’s raw response. You see what came out before it was formatted and prettied up.
If something looks off, you can trace it back. Check the query. Check the data. Check the prompt. See exactly where the problem started.
No black box. No “trust us.” Just output you can verify.
What this means for you
You don‘t need a perfect model. You need a model you can check.
With Events mode, your CFO can verify a number before putting it in a board deck. Your security team can audit the data flow without launching an investigation. Your BI team can finally stop being a search bar for the rest of the company.
And the next time the AI makes a confident, wrong guess, you’ll see it — before it becomes someone else‘s problem.
And the next time the AI makes a confident, wrong guess, you’ll see it — before it becomes someone else‘s problem.
Try it yourself
Tableau AI Assistant is free to download. You bring your own LLM key — Claude, Deepseek, or others. No vendor lock-in. No hidden costs.
We built this because we wanted a tool we could actually trust. Maybe you do too.
Tableau AI Assistant is free to download. You bring your own LLM key — Claude, Deepseek, or others. No vendor lock-in. No hidden costs.
We built this because we wanted a tool we could actually trust. Maybe you do too.
Want to see how this works on your dashboards?
Let‘s walk through your setup — no commitment, just a conversation.
By clicking the button, you agree to our Privacy Policy. We‘ll use your email only to respond to your inquiry.
Contacts