How to test AI features: a practical QA guide for LLM-powered products
AI features break the basic assumption behind most software testing: the same input produces the same output. Ask an LLM-powered assistant the same question twice and you may get two different, equally reasonable answers, or one good answer and one confidently wrong one. Testing these features well takes a different toolkit.
Why traditional test cases fall short
A classic test case checks that output exactly matches an expected value. That fails for AI features in three ways. Outputs vary from run to run. The input space is effectively unlimited, because users can type anything. And "correct" is often a judgment call: an answer can be accurate but unhelpful, or helpful but off-brand.
Start with an evaluation set
The foundation of AI testing is an evaluation set: a curated collection of realistic inputs paired with a description of what a good response looks like. Instead of exact expected strings, you record expected behaviors, such as "mentions the 30-day refund window" or "declines and points to a human agent." Build it from real user questions where you can, and cover four categories:
- Happy paths: the common questions your feature exists to answer.
- Edge cases: ambiguous, misspelled, very long or multi-part requests.
- Out-of-scope requests: things the feature should politely refuse or redirect.
- Adversarial inputs: attempts to make it misbehave, leak information or ignore its instructions.
Define what "good" means
Write a short scoring rubric before anyone starts grading. Typical criteria are factual accuracy and grounding in your source data, following instructions, tone, safety, and correct format (for example, valid JSON when another system parses the output). Human reviewers should grade a sample first. Automated, model-based grading can then scale the process, but only after you've checked that its scores agree with your human reviewers.
Test the failure modes that matter
- Hallucinations: invented facts, policies, prices or citations.
- Prompt injection: instructions hidden in user messages, uploaded documents or web pages that hijack the model.
- Data leakage: exposing the system prompt, internal tools or another customer's information.
- Over-refusal: blocking legitimate requests so often that users give up.
- Broken structured output: malformed data that crashes downstream code.
Make it repeatable
Run your evaluation set every time you change a prompt, swap or upgrade a model, adjust retrieval settings or update source documents. Track pass rates per category over time. Model upgrades in particular can quietly improve one behavior while breaking another, and a versioned evaluation suite is the only reliable way to notice.
Keep testing in production
Pre-launch testing can't anticipate everything users will try. Collect feedback signals such as thumbs up and down, escalations to human support and abandoned conversations. Review a regular sample of real conversations (with appropriate privacy controls) and feed the failures you find back into your evaluation set.
A pre-launch checklist
- An evaluation set covering all four categories above.
- A written rubric with human-calibrated scoring.
- Red-team results for prompt injection and data leakage.
- Agreed pass-rate thresholds that block a release.
- Production monitoring and a process for adding new failures to the suite.
Launching an AI feature? We build evaluation suites, run red-team testing and help teams ship LLM features they can trust. See our AI development and testing services