Know whether your AI system is right often enough to run your work
LLM evaluation with a locked test set from your own files, a pass mark agreed in writing and tracing in production. For a system we built, or one you already have.
No list price. It starts with a diagnostic at a fixed fee, and the build is quoted in writing after it.
- Refund request
- Change of date
- Missing invoice
- Duplicate record
- Out of scope
- Two bookings in one
When you need evaluation
And when you do not.
The method every system of ours includes, offered on its own. It tells you whether an AI system is right often enough to run your work before it does, and whether it is still right a month later.
You need it when
- A system someone built that nobody has measured.
- A pass mark you have to show a client, a regulator or a board.
- A change of model or vendor, and a way to know whether anything got worse.
- A system that was right at launch and seems worse now.
You do not when
- A system of ours: evaluation is already inside its price.
- A pilot with no decision riding on the result.
What evaluation includes
Part by part.
- 01
The task is written down
What a right answer is, who decides, and what a costly mistake looks like.
- 02
A test set is built
From your own files, calls or records, sized to predict production, and held locked so nobody trains or tunes on it.
- 03
A pass mark is agreed
In writing, before any measuring, with the people who will live with the result.
- 04
The system is scored
Automatically where a judge can be trusted, by your people where it cannot, and the two are checked against each other.
- 05
Production is traced
Every call logged with its inputs, outputs and cost, so a miss can be found and replayed.
- 06
Drift is watched
A sample re-scored on a schedule, with an alert when the number moves.
How evaluation is proved
A locked test set and a pass mark, agreed before the build.
- The size of the test set and what it predicts, stated with its limits.
- Agreement between automatic scoring and your reviewers, measured before automatic scoring is trusted.
- The number itself: the score against the pass mark, in a report you can hand to someone.
How evaluation is run
After launch: monitoring, updates and what you own.
- Monitoring: a drift alert when the sampled score moves, and a trace for every call.
- The test set grows as the work changes, with the old set kept so scores stay comparable.
- Handover: the test set, the scoring code and the dashboards, in your name.
What evaluation will not do
Limits we set on purpose.
- It does not score a system on a test set its builders have seen.
- It does not call a system good because a demo went well.
- It does not replace the people who know what a right answer is. It makes their judgement repeatable.
- It does not promise a result before it is measured.
Questions about evaluation
Can you evaluate a system someone else built?
Yes. We need the system, a sample of real inputs and the people who know a right answer from a wrong one. The result is a number against a pass mark you agreed, in a report you can hand on.
How big does a test set have to be?
Big enough to predict production, which depends on the task and how much it varies. We state the size and what it can and cannot tell you. This is one of the questions our research works on.
Does a model judge the answers?
Where it can be trusted, and we measure that first by checking it against your reviewers. Where they disagree, your reviewers win.
What happens when the score drops?
An alert, a trace of the calls that missed, and a fix. For a system of ours that is inside the run fee. For a system someone else built, we show you where it fails and what to do.
Talk to us about evaluation
Fifteen minutes on a call is enough to tell whether it fits your work. If it does not, we say so.