Vendors have polished demos. You need to ask the questions the demo doesn’t answer.
The demo will show you the tool working on ideal data. Your job is to figure out what happens when it doesn’t. Here are five questions that separate serious vendors from ones counting on your excitement.
“What’s the failure mode?” Every AI system has one. When it produces wrong output, what does that look like, and how would we know? Vendors who dodge this question don’t understand their own product’s failure surface. Vendors who answer it honestly are worth taking seriously.
“Can you show me a case study where this didn’t work?” Every vendor has one. If they say they don’t, they’re either lying or new. The honest answer is a story about a client whose deployment stalled, an integration that didn’t fit, or a use case that turned out to be wrong. Their willingness to tell you predicts how they’ll behave when you’re the one having a hard time.
“What’s the exit path?” If we deploy this and it doesn’t work, what does turning it off look like? What happens to our data? What integrations will we have built that stop functioning? Vendors who make it hard to leave are counting on inertia. Vendors who make it easy are confident in the value.
“Who owns the ongoing improvement?” AI systems degrade if nobody’s watching. Whose job is it to notice the quality dropping — yours or theirs? If it’s yours, what tools does the vendor provide? If it’s theirs, what’s the SLA? “We continuously improve the model” is not an answer. “We report accuracy metrics monthly and you have veto power on model updates” is.
“What’s the baseline you’re measuring against?” Vendors love to quote productivity improvements. Improvements against what baseline? Their baseline is not your baseline. Ask them to help you build a baseline for your specific processes before deployment, so you can measure honestly six months later. Vendors who refuse are afraid of the measurement.
Bonus question if the demo went well: “Can we run this on real data with a small team for 30 days before signing?” A pilot period isn’t optional for meaningful AI deployment. Vendors who won’t offer one are asking you to pay for the pilot yourself.
Interviewing vendors this way takes an extra hour per conversation. You’ll do fewer deployments, and the ones you do will land. That’s the tradeoff, and it’s the right one.
If you’re not sure your business is ready to even have these conversations productively, the diagnostic is a good first check.