TL;DR
- Test AI support vendors on your own past tickets, not a scripted demo.
- Every answer an AI support agent gives should show the source it came from.
- Refund caps and policy limits belong in the workflow, not the prompt.
- Compare AI support pricing on cost per resolved ticket.
An AI customer support RFP is a written set of questions that lets you compare vendors on equal terms. The strongest ones test accuracy, answer sources, actions, security, human handoff and the full cost per resolved ticket.
Getting this right matters more each year. Forrester's 2025 CX Index found that customer experience quality in the US and Canada fell for a fourth straight year, to a new all-time low. Meanwhile, MarketsandMarkets projects the AI for customer service market will grow from $12.06 billion in 2024 to $47.82 billion by 2030, so the vendor list keeps getting longer.
Much of a support queue also needs more than a help article. Support teams tell Brainfish that articles answer only 20 to 30% of their tickets, and the rest require someone to check a system and take action.
Most AI support demos look great because they use easy questions. The trouble shows up after launch, when customers ask messy questions, your product changes, and someone asks for a refund the AI isn't sure it can give. The 30 questions below help you find those gaps first, and you can paste them straight into your procurement doc.
How do you run an AI support vendor evaluation?
Send the questions in writing first so every vendor answers the same list. Then pick 10 to 20 tickets your team handled last quarter, including the messy ones, and ask each shortlisted vendor to run them live.
Score each answer from 1 to 3. A 1 means the vendor couldn't answer, a 2 means they answered but couldn't show it, and a 3 means they showed it working on your data.
Here's what good and bad answers look like in each area.
What should you ask about accuracy?
A fast wrong answer costs more than a slow right one, because the customer acts on it and comes back frustrated. These questions show how accurate a vendor's agent is, and how they prove it.
1. How do you measure answer accuracy, and can we see results for a customer like us? Accuracy means different things to different vendors. Ask how it's measured, who grades it, and whether the number comes from live traffic or a test set.
2. Will you test the agent on our past tickets before we sign? Hand over a sample of past tickets, including the edge cases your team argued about. A vendor that trusts its accuracy should agree without a long negotiation.
3. What happens when the agent doesn't know the answer? You want an agent that admits it isn't sure and routes the question, rather than guessing. Ask to see exactly what it does when it has nothing solid to go on.
4. Does the agent handle the same request the same way every time? Ask the vendor to run one tricky request five times in a row. If the outcomes change from run to run, picture that across thousands of customers.
5. How do you define a "resolution"? Some vendors count any conversation that didn't turn into a ticket. Others count only conversations where the problem was actually solved, so get the definition in writing before you compare numbers.
6. How do we review and correct answers after launch? Look for a way to see every conversation, rate answers, and fix the source of a bad one. If every correction needs a ticket to the vendor, mistakes stay live longer.
What should you ask about grounding and knowledge?
An AI support agent can only be as accurate as the content it reads. These questions show whether answers come from your material or from the model's best guess.
7. Does every answer show the source it came from? Sources let customers and your team check an answer in seconds. Ask whether they're visible to customers, to internal reviewers, or both.
8. Which content sources can you connect, and how often do they sync? Your help center is only the start. Many B2B teams also keep answers in product docs, Notion, Confluence, Google Drive and old tickets, so ask how often each one syncs.
9. How do you catch content that has gone out of date? Products often change faster than help centers do. Ask whether the vendor flags stale or conflicting articles, or whether your team has to find them by hand.
10. Can answers change based on a customer's plan, region or role? A customer on a starter plan shouldn't be walked through an enterprise feature they don't have. Ask how the agent tells customers apart and how that changes what it says.
11. Do we need to rebuild our knowledge base or switch helpdesks first? Some tools need content restructured or a helpdesk migration before they work. That adds time and risk, so ask for every prerequisite to be listed in the RFP response.
What should you ask about taking action in your systems?
Answering how-to questions is the easy part. Refunds, cancellations and account changes are where AI support agents either earn their place or send the ticket back to your team.
12. Which ticket types can the agent fully resolve, not just answer? Ask for a list of the specific actions the agent can complete, such as refunds, cancellations, plan changes, account unlocks and log checks. Then ask which of those are live with customers today.
13. Where do policy limits live, in the prompt or outside it? A prompt is an instruction a model can misread. A refund cap written into the workflow itself is a rule the AI can't talk its way around.
14. Can we test a workflow against past tickets before it goes live? You want a way to replay past tickets through the workflow, plus version history and rollback. Shipping straight to customers and watching what happens is not a test plan.
15. Who builds and maintains workflows, and how long does it take? If every change needs your engineering team, workflows will sit in a backlog. Ask whether your support ops team can build and edit them, and what a typical build time looks like.
16. What gets logged when the agent takes an action? For every action, you should see the steps taken, the systems touched and the data used. That record matters for audits and for explaining a decision to an upset customer.
What should you ask about security and compliance?
Security review is often one of the last gates before a contract is signed. Ask these questions early so it doesn't hold you up at the finish line.
17. Which certifications do you hold, and can we see the reports? Ask for SOC 2 Type II and ISO 27001 at minimum, plus the vendor's GDPR approach if you serve EU customers. A trust portal with ready-to-share reports can save weeks.
18. Is our data used to train your models or anyone else's? The answer should be a clear no, written into the contract. Ask about the underlying model provider too, since your data may pass through them.
19. Where is our data hosted, and can we choose the region? Many B2B companies have residency requirements in the US, EU or Australia. Confirm that both storage and processing stay in that region.
20. How is sensitive customer data handled in conversations? Ask whether personal details are masked, how long transcripts are kept, and whether you can delete them on request.
21. Do you support SSO, role-based access and an admin audit trail? Your security team will want to know who changed a workflow or an answer, and when. Ask to see the admin logs, not just a yes on a questionnaire.
What should you ask about handoff to your team?
No AI support agent resolves everything. How it hands over decides whether the customer's next minute is easy or frustrating.
22. How does a customer reach a person, and how many steps does it take? Ask to see it live. A bot that keeps asking what the customer needs after they've asked for a human is one of the fastest ways to lose their trust.
23. What does our team receive at handoff? The best handoffs include the full conversation, every check the AI already ran, and a suggested next step. Your team shouldn't have to ask the customer to start over.
24. Which helpdesks do you integrate with today? Get a specific list of what's live and what's on the roadmap. A promise to integrate with everything can mean a generic API your team has to wire up.
25. Can we set our own escalation rules? You may want certain topics, accounts or frustrated customers sent straight to a person. Ask how those rules are set and who on your team can change them.
What should you ask about pricing?
AI support pricing varies a lot between vendors, which makes side-by-side comparison hard. These questions get you to a number you can actually compare.
26. What exactly are we paying for: seats, conversations, resolutions or actions? Each model rewards different behavior. Per-conversation pricing, for example, charges the same whether the customer's problem got solved or not.
27. What will we pay per resolved ticket at our current volume? Give every vendor your monthly ticket count and ask for a full cost estimate. Divide that by expected resolutions and compare it to what each ticket costs you today.
28. What happens if we go over our usage limit? Ask about overage rates, hard caps, and whether the service stops or keeps billing. Knowing this upfront avoids a surprise invoice later.
29. What's included in implementation, and is it billed separately? Onboarding, content cleanup, workflow builds and integrations can each carry their own fee. Get them listed so your first-year number is complete.
30. What are the contract length, exit terms and data return policy? Ask what happens to your data and setup if you leave. A vendor confident in its results shouldn't need a long lock-in to keep you.
How does Brainfish answer these questions?
Brainfish builds AI support agents designed for B2B, where a wrong answer costs a customer relationship. It answers customers from your docs and takes action in your systems, with accuracy built into both.
- Answers you can check. Every answer carries a source, and every conversation is scored in Brainfish analytics.
- Rules that hold. Ballet by Brainfish runs your processes as workflows written in code, so refund caps and approval limits sit outside the prompt.
- Tested before launch. Workflows are simulated against your past tickets, versioned, and easy to roll back.
- Security from day one. SOC 2 Type II, ISO 27001, GDPR, hosting in the US, EU or AU, SSO, and no training on your data. Details are on the Trust & Safety portal.
- Handoff with the work done. When a ticket needs a person, it lands in Zendesk or Salesforce with every check already run and a suggested next step.
The difference shows up when the same request runs more than once. In a side-by-side example on the Brainfish homepage, one refund request runs five times. A prompt-only agent handles it correctly twice, while the Brainfish workflow gets it right all five times.
Huntress, a cybersecurity company protecting more than 250,000 businesses, took this approach. Before any customer saw Brainfish, the support team tested it in Slack with the kinds of questions customers ask, rating every answer and fixing each weak spot.
"It's always trust but verify. If you're the source of truth, you have to be sure the information you're putting out is correct." Adam Wasmus, VP of Global Support, Huntress
Huntress reached a 51% prevention rate across about 5,000 monthly queries, with CSAT held at 99%. It got there with no knowledge base rebuild and no forced migration.
See Brainfish pricing for plan details, or bring your hardest ticket to a Brainfish demo and watch the workflow get built live on your own systems.