How we test. Published so you can check it.

1. 01

   Realistic scenarios, built from your business

   Generic benchmarks tell you how a vendor performs on someone else’s questions. We build test scenarios from your policies, your product, and your real support tickets, including the questions your team finds hardest, the edge cases, and the customers who push back.

   A scenario is a full conversation, not a single prompt: a customer with a goal, the context the agent would have in production, and a defined correct outcome.

2. 02

   Identical conditions for every vendor

   Every vendor gets the same knowledge base, the same policies, the same scenarios, and the same grading. Where a vendor needs configuration, we follow its documented setup and record exactly what we did, so differences in results reflect differences in the product, not in the test.

3. 03

   Checkable outcomes

   We grade what can be checked against your material, not impressions of tone. Each scenario is scored on four outcomes:

   Correct resolution

   Did the customer end up with the right answer or the right action, the one your team would have given?

   Policy adherence

   Did the agent follow your written policies, including the inconvenient ones: refund windows, identity checks, exceptions that need approval?

   Escalation

   Did it hand off to a human when it should, and avoid handing off when it didn’t need to?

   Invented information

   Did it state anything that isn’t true or isn’t in your material: a made-up policy, a feature you don’t have, a promise you can’t keep?

4. 04

   Expert review of grading

   Automated grading is fast but can be wrong. Grading decisions are reviewed by people who know customer support, and we check a sample of every grader’s calls. Where reviewers disagree, we resolve it and record how, before anything goes into your report.

5. 05

   Private, held-out test sets

   Vendors never see our scenarios in advance, and we keep a portion of every test set private and unused until the final run. A vendor can’t tune its product to a test it hasn’t seen.

6. 06

   Results are dated and scoped

   AI products change quickly. Every result states the vendor version and configuration we tested, the test date, and which scenarios it covers. We don’t generalize a result beyond what we tested, and we say so when a finding may not hold after a vendor update.

7. 07

   How we handle your data

   We use your tickets, policies, and product material only with your permission and only for your evaluation. Customer tickets are anonymized before they’re turned into test scenarios: names, contact details, and account information are removed.

   We don’t share your material with other clients, and vendors see only what they need to run the test.

A note on stage. Evidyne is taking on its first evaluation clients. This page describes how we run evaluations, not results we’ve already published. As we learn, we’ll update it and keep a dated record of what changed.