A system should prove itself on your files before it touches live work. Not on a demo, not on a sample the vendor brought, but on the records your team handled last month, with the answers your team gave. This guide describes the sequence we use where more is at stake: a locked test set, a threshold agreed before the first run, shadow mode beside the team, an assisted launch, and a rule for when a category can run on its own.
The sequence is slower than switching a system on. It is also the reason the team trusts what it does. A standard system takes the short form of it: the locked test set, the pass mark agreed in writing, and a page of limits, with everything outside that page going to a person.
Build a locked test set
A test set is a collection of real cases from your records with the correct answer written beside each one. For a call-handling system, it is a set of recorded calls and the booking or note each call should have produced. For a document system, it is a set of documents and the fields a person extracted from each. For an invoice check, it is invoices with the decision a person made and why.
Three rules make a test set worth having.
- The cases are real and recent. Pull them from the last few months of work. Include the awkward ones: the caller who changed their mind twice, the scanned form with coffee on it, the invoice with two purchase orders. A test set made of easy cases proves nothing.
- The answers are written by people who do the work. Not by the vendor, and not by the owner unless the owner does the work. The person who clears the inbox every morning knows what a correct answer looks like.
- It is locked. Once the answers are agreed, nobody changes the set. The system is not tuned on it, and cases are not quietly removed because they are hard. A locked set is the only way to compare one week's result with the next.
How large should it be? Large enough that every category of work you care about appears several times. That is a judgement about your work, not a fixed number. If a category is rare but costly, add more of it on purpose.
Keep a second, smaller set that the builders never see. It is the check on the check.
Agree a threshold in advance
Before the first run, decide what good enough means, and write it down. Decide it per category, because a system can be ready for routine bookings and not ready for cancellations.
Two measures matter most. The first is how often the system's answer matches the person's answer on the locked set. The second is how often the system stops and asks instead of answering. A system that stops often is slower but safe. A system that never stops is either very good or has not been given permission to say "I do not know". You want to see both figures before you trust either.
Set the threshold with the person who will review the system's work. They know what a mistake costs. Then resist moving it after the results come in. If the system misses the line, the system changes, not the line.
Decide what passing looks like before you see the result. Afterwards, every number looks like it could be enough.
Shadow mode beside the team
In shadow mode the system runs on live work and does nothing with it. The team keeps working exactly as before. The system produces its answer to every case, the answer is stored, and nobody acts on it. At the end of each day, or each week for slower work, the two sets of answers are compared.
Shadow mode finds the things a test set cannot. It finds the new kind of request that started last Tuesday. It finds the supplier whose invoices changed format. It finds the step where the team does something the rule never mentioned.
Run it long enough to see a full cycle of the work. For most workflows that means a month end, a busy week and a quiet one. Every disagreement between the system and the team goes into a log with three columns: what the system said, what the person did, and which one was right. Sometimes it is the person who was wrong, and that is useful to know too.
Fix patterns, not single cases. Three disagreements of the same kind point to a rule that needs work. One odd case is an odd case.
Assisted launch
When shadow mode meets the threshold, the system starts to act, and a person approves every action before it takes effect. The system drafts the booking; the receptionist confirms it. The system fills the fields; the bookkeeper checks them before they post. The review step is the launch.
Two things are measured during the assisted period. How often the reviewer changes something before approving, and how long the review takes. If the reviewer changes little and the review is quick, the category is a candidate for running on its own. If the reviewer changes a lot, the system goes back to shadow mode for that category while the rule is fixed.
Designing this approval step well is its own subject. We cover what a reviewer should see and how confidence should be shown in A person in the loop.
When to automate a category
A category can run without review when three things are true at the same time: it met the threshold on the locked set, it held that result through shadow mode, and the reviewer rarely changed anything during the assisted period. The table shows how the decision usually falls.
| Category | Locked set | Shadow mode | Reviewer changes | Decision |
|---|---|---|---|---|
| Routine booking changes | Met the line | Held | Rare | Run on its own |
| New client intake | Met the line | Held | Occasional | Keep the review |
| Cancellations with a refund | Met the line | Mixed | Frequent | Back to shadow mode |
| Complaints | Below the line | Not run | Not run | Stays with people |
Automating a category is not permanent. If the reviewer's spot checks find a change in the work, the category returns to review. Nothing is lost by moving it back.
Some categories never leave review, by design. Anything that commits money, declines a client or touches a legal obligation can stay with a person for as long as you like. A system that drafts those and waits is still doing most of the work.
After launch
The locked set does not retire. It is run again after every change to the system, every change to a model, and every change to the software it connects to. The result is compared with the last one. If a change makes the system worse on the locked set, the change does not ship.
Keep the shadow log too. A month of disagreements, read in one sitting, tells you more about how the work is really done than any process document.
If you are starting from the beginning, the first step is choosing a workflow where all of this is possible: one channel, one rule, one record. That is covered in How to choose the first workflow.