What does a correct AI answer really cost? Webinar recap

By Juraj Slota | 5 min read

In short: We asked Claude models 21 finance questions on NetSuite data, once over a raw SQL connection and once over Data to AI. The cheapest model on Data to AI matched the most capable model on raw SQL at 24x lower cost per question. In the webinar, Radim Marek showed the three ways the models went wrong without it: counting rows twice, mixing currencies, and answering a question the data could not answer.

On 19 November, Petr Nemeth (Dataddo's founder) and Radim Marek, who leads Data to AI, walked through our latest benchmark and the mistakes behind its numbers. Here is what they covered.

Watch the full recording, or download the benchmark whitepaper for the full methodology.

Why finance data

Petr explained the choice simply: ask for EBITDA and there is exactly one right answer. That makes finance the cleanest place to tell a correct answer from a wrong one.

The setup:

  • A NetSuite company group derived from real-world business data: 200,000 transactions and 2,000 customers, invoiced in several currencies.
  • 21 business questions, each asked three times per setup. Three of them cannot be answered from the data, on purpose.
  • Three Claude models: Haiku 5.5, Sonnet 5 and Opus 5.5.
  • The same data every time. The only difference was how the model reached it: a raw SQL connection, CSV exports, or Data to AI.
  • The answer key was fixed before the run. A good outcome is a correct answer, or a correct refusal.

The headline: the connection mattered more than the model

  • A small model on Data to AI matched a frontier model on raw SQL. Haiku 5.5 on Data to AI reached 87.3% good outcomes. Opus 5.5 on raw SQL reached 81%. On 63 answers each, read that as "matched", not "beat".
  • At 24x lower cost per question: $0.0045 against $0.108.
  • Fewer confidently wrong answers with every model. Haiku 5.5 went from 14 to 8, Sonnet 5 from 19 to 8 (58% fewer), Opus 5.5 from 12 to 3, each out of 63.

Petr's takeaway: if you don't plan to run a frontier model on every question, money spent on the data layer can get you comparable answers for a fraction of the price.

Three confidently wrong answers, and how Data to AI handled them

A confidently wrong answer is one you can't tell is wrong without significant effort: precise, well explained, and off. A crash is easy to spot. A plausible number gets built on. (We explain the idea in more depth in What is a confidently wrong answer? blog post)

Radim walked through three questions from the benchmark where models connected straight to the database got it wrong, and what Data to AI did differently.

Clip: What is a confidently wrong answer?

1. Duplicate rows. How many units did we invoice in 2025? The data held 13 invoice lines twice. On raw SQL, 12 runs in a row answered 16,836 instead of 16,810, and the most capable model overcorrected to 15,092 by dropping freight and assembly lines. Data to AI blocks a plain sum on invoice lines, removes duplicates by line ID, leaves out voided and credited units, and returns 16,810.

2. Currency IDs. What share of invoiced value falls in each currency? NetSuite stores currencies as numeric IDs. One model gave the right maths with unreadable labels ("Currency 4"); another applied the exchange rate backwards and put 88.47% of value in Czech koruna, whose real share is 1.34%. The same model on Data to AI got it right, because each ID maps to a verified ISO code and conversion runs in the base currency by fixed rules.

3. A question the data can't answer. What was our best-selling product in 2025, by product name? There are no product names in the data. On raw SQL, models picked a subproject column instead and named three different "best sellers" across six runs. Every Data to AI run declined, said why, and gave the top item by ID instead.

How it works: Resolve, Ask, Stop

The key difference from text-to-SQL: the business rules are not loaded into the AI's prompt in the hope it follows them. They sit in a separate, predictable layer, and that layer builds the query. The AI never writes free-form SQL; it chooses from defined entities.

Three frameworks cover the failure modes:

  • Resolve. Applies the semantic model to what goes in and what comes out: the right exchange rate, the difference between invoice lines and invoice headers, the facts about each account.
  • Ask. When a question could mislead (an ambiguous name, a field known to confuse), the assistant asks you to clarify instead of picking a meaning. This works with models that support it.
  • Stop. Declines operations that can't be done safely, such as joins that multiply rows, sums of percentages, or totals across currencies, and explains why.

And every number comes with a receipt. A SQL query hands the AI what is effectively a spreadsheet. Data to AI hands it the table plus what is known about it: where it came from, how current it is, what it includes. That lets the model make better decisions without guessing.

Radim's summary: accuracy needs semantic guardrails, and the guardrails have to be enforced, not suggested.

Questions from the audience

  • What is a confidently wrong answer? An answer you can't tell is wrong without real effort, and one that can send a whole investigation in the wrong direction. It matters most with finance data, where a number is either right or it isn't.
  • Did you test CSV exports? Yes. The whitepaper includes a CSV-exports setup with Sonnet 5: 24 confidently wrong answers out of 63, against 8 on Data to AI. ERP systems such as NetSuite usually offer a SQL interface, so raw SQL was the main comparison.
  • Does my data stay in the EU? Dataddo offers EU data residency, and Data to AI works with any MCP-compatible model, including models hosted in Europe.
  • Can Data to AI use the semantic model we already have? That is the goal. Rather than create another standard, Data to AI is built to work alongside semantic models you maintain in tools like Microsoft Fabric or Looker.
  • What is the most common mistake companies make when they first use AI on business data? Radim: "Data layer, and the way it's connected, is usually an afterthought."

Run the test on your own data

You don't need a data team to see whether your AI guesses. Section 6 of the benchmark whitepaper is a 20-minute test: six questions to ask the assistant your team already uses on business numbers.

Then try the same questions through Data to AI. Start free, learn more on the Data to AI page, or watch the full recording.

Comments