Public benchmarks are useful. They tell you roughly where a model stands. They don't tell you whether it will handle your contracts, your codebase or your support queue. For that, you need a suite of your own.
Why public scores aren't enough
A benchmark measures one kind of work under one set of conditions. Your work has its own vocabulary, its own edge cases and its own idea of a good answer. Two models with similar public scores can behave very differently on it, and the difference only shows up after you have switched.
There is a second problem. Models change under the same name. A model that passed your tests in the spring may answer differently in the autumn. Without your own suite, you find out from users.
Start small and real
The best first suite is small. Collect a few dozen tasks from real work: requests your system actually received, with the answers your best people would accept. Include the cases that went wrong before. A short list of real tasks beats a long list of invented ones.
Write down what a good answer looks like for each task. Sometimes that is an exact value. More often it is a short checklist: cites the right clause, keeps the tests passing, doesn't invent a number.
Grade what matters
Measure the things that change decisions:
- Quality: did it meet the checklist?
- Cost: what did the run cost, including retries?
- Failure: when it failed, how? Wrong, incomplete, or confidently wrong?
- Speed: how long did a person wait?
Automated grading works for much of this. For the rest, a person reviews a sample. The goal isn't a perfect score. It is a number you trust enough to act on.
Run it on every change
A suite earns its keep when it runs automatically: on a new model, a new prompt, a new tool or a provider update. Treat a falling score like a failing test. Someone looks at it before it reaches users.
A model change should be a decision, not a surprise.
What you get
With a suite in place, model choices stop being arguments about impressions. You can try a cheaper model and know what you gave up. You can adopt a new release and know what you gained. And when something breaks, you have the tasks to show exactly where.
We build these suites as part of every system we deliver, and we run the same kind of suite on our own work every day.