I Built an AI Tool That Refuses to Make Up Numbers
Survey Insight Skills is a small open-source toolkit. Building it taught me something about using AI without getting burned.
Over the last few months, analyzing survey data was part of my work. Nothing exotic. A CSV full of answers, a stakeholder waiting for a clear picture, and me trying to turn one into the other before the meeting.
The obvious move in 2026 is to hand the file to an AI and ask: “How satisfied are users, and does it depend on company size?” You get a confident, well-written answer in seconds. That is also where the trouble starts.
The problem nobody puts on the box
Language models are built to produce plausible text. That is their whole talent. But a percentage that looks right is not the same as a percentage that is right, and the model cannot always tell the difference either.
Ask one to read a spreadsheet and compute satisfaction by company size, and it will happily give you 63%. Sometimes it actually counted the rows. Sometimes it pattern matched its way to a number that felt about right. From the outside, the two answers are identical. Both are fluent, both are formatted, both sound sure of themselves.
An answer you cannot check is not an insight. It is a rumor with a decimal point.
For survey data, this matters, because the whole job is trust. If I put “larger companies are less satisfied” in front of a product team, people make decisions on it. I would like those decisions to rest on arithmetic, not on vibes.
Building it the other way around
So I built the tool backwards from how people usually reach for AI.
The model never touches the math. It reads your question, decides what to analyze, and writes the report at the end. Everything in between, every count, every percentage, every cross tab, is done by a plain Python engine. No model in the loop for the numbers.
Every figure in the output is computed, not generated. When there is a group difference, the engine runs an actual significance test and reports an effect size, so a random blip in a small subgroup does not get dressed up as a finding. If the sample is too thin to trust, it says so instead of inventing confidence.
The result is boring in the best way. The model does what it is genuinely good at, understanding a messy question and writing a clear answer. The engine does what it is good at, the same arithmetic every single time. Neither one is asked to fake the other’s job.
The part that is not about surveys
This is the lesson I keep relearning with AI, and it is why I wrote this down.
The useful way to work with these models right now is to let them reason, and to never let them be the source of truth for anything you can verify by other means. Give a model a calculator and it stops needing to pretend it is one. Give it a search tool and it stops guessing what a document says. The failures happen when we ask the model to be the reasoning and the fact at the same time.
So the interesting design question is almost never “can the model do this”. Of course it can, it will do anything you ask, that is half the problem. The better question is “what should this model never be trusted to do on its own, and how do I build around that”. Answer that well and the AI part gets quietly reliable. Skip it and you ship confident nonsense.
None of this is a knock on the models. It is just knowing which tool holds which job. A brilliant analyst with no calculator still should not be reading numbers off the top of their head.
Try it, and tell me how you do this
I open-sourced the whole thing. It is a set of slash commands for Claude Code, it runs on the included sample data out of the box, and it works with any survey CSV, in English or German. No dependencies, no setup beyond copying a folder.
Repo: github.com/kayspiegel/survey-insight-skills
I mostly built it for myself, which is the honest reason it exists at all. But now I am curious whether it is useful to anyone else. So if you also work with survey data, I would like to know two things: where it breaks for you, and how you handle this today. Spreadsheet gymnastics? A paid tool? Something you rigged up yourself?
Tell me. I have been quiet for a while, and this feels like a good thing to start writing about again.