Confidence matters because, according to OpenAI's hallucination research, standard training and evaluation reward models for guessing rather than admitting uncertainty. On OpenAI's SimpleQA test of short factual questions, its o4-mini model abstained on just 1% of questions and answered 75% wrong.
A different model, gpt-5-thinking-mini, abstained on 52% of questions and answered 26% wrong, with slightly lower accuracy (22% vs. 24%). OpenAI's takeaway: guessing when uncertain can lift accuracy scores but produces more errors. In support, a confident wrong answer about a refund policy costs more than an honest handoff.
Huntress, a cybersecurity company protecting more than 250,000 businesses, checked answer quality before going live: its support team asked Brainfish real questions and rated each response. Huntress now reports a 51% ticket prevention rate with CSAT held at 99%, according to the Huntress customer story.
How does an AI confidence score work?
Methods vary by vendor. Common signals include:
- Model probability. How likely the model thinks its own words are, based on the probabilities it assigns while generating the answer.
- Retrieval match. How closely the documents the AI found match the question. A weak match is a warning sign even when the answer sounds fluent.
- Answer evaluation. A second check that grades the finished answer against its sources, sometimes using another model as a reviewer.
Raw model probabilities aren't always reliable on their own, which is why calibration matters.
What does a "calibrated" confidence score mean?
A calibrated score means what it says. If an AI agent gives 100 answers a confidence of 0.9, about 90 of them should be right.
In a peer-reviewed study presented at the 2017 International Conference on Machine Learning, Cornell University researchers found that modern neural networks are poorly calibrated compared with models from a decade earlier, and often overconfident. The study tested image and text classifiers rather than chatbots, but the lesson still applies: a raw score isn't a guarantee.
The practical step is to test scores against real outcomes, like past tickets with known answers, before you trust a threshold.
How do support teams use confidence thresholds?
A confidence threshold is the cutoff that triggers a different action. Below it, the AI stops answering on its own. Here's one way a team might split the range:
The exact cutoffs depend on your ticket mix and how costly a wrong answer is. The low end should feed a clean AI-to-human handoff, not a dead end.
Why isn't a high confidence score enough on its own?
Because an AI can be confident and wrong. If the help article it retrieved is out of date, the AI can repeat the old answer with high confidence, since nothing in the source looks uncertain.
That's why a score needs evidence behind it. Every answer should point to a source a person can check, which is the core idea behind grounded answers. Some decisions shouldn't depend on confidence at all: refund caps, account deletions, and identity checks belong in fixed rules the AI can't talk its way around.
Brainfish CEO Daniel Kimber describes what to look for in Brainfish's AI support guide:
"By 2026, most vendors can demo a good model. The differentiator is whether the operating model underneath is real: citations you can audit, escalation that carries context, and analytics that tell you which articles are actually resolving questions. When you can see what the agent retrieved and what happened next, you can improve it like any other system." Daniel Kimber, CEO & Co-Founder, Brainfish
What's the difference between a confidence score and an accuracy rate?
A confidence score is an estimate about one answer, made before anyone knows if it's right. An accuracy rate is a measured result across many answers, calculated after the fact. Brainfish customer Gradeo, an online assessment platform for schools, reports 89% AI answer accuracy in its Gradeo case study.
You need both. Confidence routes each conversation in the moment, and accuracy tells you whether those routing decisions were sound.
How does Brainfish score its AI agent's answers?
Brainfish builds AI support agents for B2B teams and lets your team see the score behind every answer. In Brainfish analytics, every conversation is scored on relevance, evidence, and policy checks, with the sources attached.
Analytics also surface knowledge gaps, so your team can spot a weak answer, fix the content, and approve the change before customers see it. For actions like refunds, Brainfish workflows keep limits in code, so anything outside your policy stops for a person.
Huntress's support team watches Brainfish analytics continuously, checking where each answer is sourced from.
Want to see Brainfish on your own tickets? Book a demo and bring a real ticket, or see Brainfish pricing.