A language model takes text in and gives text out. On its own it cannot read your calendar, does not know your prices and has no idea who your clients are. It will answer a question about any of those anyway.
A business does not need answers to questions. It needs jobs done: the call answered and the appointment booked, the document read and filed, the follow-up sent and the reply tracked. The distance between a model that answers and a system that does a job is where nearly all the work is.
This post defines the term, because we use it often and it means something specific.
A definition
An AI system is a model plus everything that lets it do a job reliably inside a business: the data it reads, the tools it can use, the rules that limit it, the people who approve its work and the monitoring that shows whether it is working.
The model is one component. It is not the product.
The model is the smallest part
In 2015 a group of engineers at Google published a paper with a memorable diagram. It drew the machine learning code as a small black box in the middle of the picture. Around it, far larger, were the boxes for the rest of the system: data collection, verification, configuration, serving infrastructure and monitoring.
Their point was that the model is a small fraction of a working system, and that the surrounding parts are where the long-term cost lies. The paper was written before large language models. The observation applies to them without change. The models have become far more capable. They have not become a larger share of what it takes to run one dependably.
The six parts
1. The model. The component that reads and writes language. Chosen for the job: a call needs speed, a contract review needs care. It can be changed later without rebuilding everything else, if the rest is built well.
2. The context. What the model is given to read at the moment it works: the relevant client record, the right policy document, today's schedule. A model with the right context gets the answer from the page. A model without it makes one up. Most of the quality of a system is decided here.
3. The tools. The connections that let the system act: the calendar, the phone line, the document store, the accounting package. Each one comes with a decision about what the system may do there. Reading is granted freely. Writing is granted with care.
4. The checks. Rules that sit outside the model and do not depend on it behaving. An amount must match a purchase order. A date must be in the future. A reply must not go to an address that is not on file. These are ordinary code, and they catch what a model, however good, will sometimes get wrong.
5. The people. The step where a person looks and decides. It is not a courtesy. It is where judgement and accountability enter. Deciding which actions need approval, and what the approver sees, is a design problem of its own. We wrote about why it is harder than it looks in Automation bias: why people stop checking the machine.
6. The record. Tests before launch, on real examples. Logs after launch, in plain language, of what came in, what the system did and why. Measures that show whether it is getting better or worse. Without this, nobody can say whether the system works. They can only say it has not visibly failed.
A chatbot, an assistant and a system
The three are often confused. They differ in where they live and what they can touch.
| Where it lives | What it can touch | Who checks the work | |
|---|---|---|---|
| A chatbot | A window on a website or an app | Nothing. It replies with text | The person reading |
| An assistant | Beside one person, in their tools | That person's files and messages, when asked | That person, every time |
| A system | Inside a process, across the tools the business runs on | The records and actions the process needs, within set limits | Built-in checks, and a named approver at the steps that matter |
A chatbot is finished when it has replied. A system is finished when the appointment is in the calendar, the confirmation has gone out and the record shows who approved what.
What "production" adds
A demonstration shows that something can work. Production means it does work, on the hundredth day as on the first.
It copes with bad input. The scanned page is upside down. The caller changes their mind halfway through. The spreadsheet has a new column. A production system notices and has somewhere to send the case. It does not carry on as if nothing happened.
It fails where someone can see. When it cannot do something, it says so and hands over. The worst failure is the quiet one, where a wrong result goes into a record and is found a month later.
It can be inspected. Anyone responsible for it can open the log and read what happened to a given case, without needing an engineer to translate.
It is maintained. The tools it connects to change. The business changes. The models change. Somebody has to own the system after launch, or it decays.
Questions worth asking
Whether you are buying a system, building one or being sold one, these questions sort the serious proposals from the rest.
- What does it read, and where does that information go? You should get a list, not a reassurance.
- What can it change, and what stops it changing anything else?
- What happens when it is unsure? There should be a designed path, and it should lead to a person.
- How was it tested, and on whose examples? Tests on your own past cases mean something. Tests on a generic set mean little.
- Can I read the log? Ask to see one.
- Who looks after it in six months?
A proposal that answers these plainly describes a system. One that talks only about the model describes a component.
Where to go from here
Our systems pages describe the twelve kinds of system we build, by the job each one does. The approach page covers how one is scoped, tested and launched. If the question is where to begin, the guide to choosing the first workflow is the place to start, and security and trust explains how client information is handled along the way.
References
- Sculley, D., Holt, G., Golovin, D., and others (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28. NeurIPS proceedings
