Ask how an AI system is kept safe and the answer is usually the same four words: a person reviews it.
It is a good answer. It is also the part of the design most likely to fail without anyone noticing, because it depends on something people are known to be bad at. The research on this is older than the personal computer, and it is unusually consistent.
What automation bias is
Automation bias is the tendency to take the output of an automated system as a substitute for looking. Kathleen Mosier and Linda Skitka gave it the name in the 1990s, after studying how pilots used cockpit decision aids.
It produces two kinds of error.
- Errors of omission. The person misses a problem because the system did not flag it. The warning light is off, so nothing is wrong.
- Errors of commission. The person follows a recommendation that is wrong, even when other information in front of them contradicts it.
In a simulated flight task, Skitka, Mosier and Burdick compared people working with an automated aid against people working without one. When the aid failed to prompt, or gave a wrong instruction, the people with the aid made errors that the people without it did not. The automation had not been added to their judgement. It had replaced part of it.
It is not a beginner's mistake
The comfortable reading is that this happens to people who are careless or new. The research does not support it.
Raja Parasuraman and Dietrich Manzey reviewed the evidence in 2010 and drew three conclusions that matter for anyone designing a review step. Automation bias appears in experts as well as novices. It appears in teams as well as individuals. And it cannot be prevented by training or instructions alone. Telling people to check does not, by itself, make them check.
A 2012 systematic review by Kate Goddard and colleagues, covering 74 studies of decision support in health care, aviation and other fields, adds detail. The bias was more common in people new to a task, but experienced users were not immune. What reduced it was training, making users accountable for their decisions, and design: where the advice sat on the screen, whether it came with an up-to-date confidence level, and whether the system offered information or a ready recommendation.
Why watching is hard
Underneath automation bias is an older finding about attention.
During the Second World War, the Royal Air Force noticed that radar operators missed more signals toward the end of a watch. The psychologist Norman Mackworth was asked to find out why. He built a plain clock face with a single hand that moved in steps. Every so often the hand jumped two steps, not one. Participants watched it for two hours and pressed a key when they saw a double jump.
Detection dropped within the first half hour and kept drifting down after that. Mackworth published the result in 1948. It has been repeated in many forms since, and it has a name: the vigilance decrement.
People are poor at watching for rare events in a stream of ordinary ones. It is not a matter of motivation. The radar operators knew what was at stake.
A review queue in which nearly every item is correct is a Mackworth clock.
The irony of reliable systems
In 1983 the psychologist Lisanne Bainbridge published a short paper called "Ironies of Automation". Its argument has aged well.
When a task is automated, she wrote, the person is left with two jobs: monitoring the automation for the rare occasion it fails, and taking over when it does. The first is the job people do worst. The second requires skills that fade from lack of use. The better the automation, the less practice the person gets, and the harder the moment of takeover becomes.
Applied to an AI system, the irony is direct. A system that is right almost every time trains its reviewer to approve. Each correct result is a small lesson that looking was unnecessary. The reviewer's trust rises to match the system's usual performance, which is exactly the wrong setting for the unusual case.
What it looks like in a business
None of this needs a cockpit. It happens at a desk.
A model drafts replies to client emails and a staff member approves them. In the first week every draft is read. By the fourth week the approve button is pressed on the strength of the first line.
A model reads invoices and fills in the supplier, the amount and the account code. The bookkeeper's job is to confirm. The fields are almost always right, so the eye stops landing on them.
A model summarises an intake call and the summary goes into the file. Nobody replays the call, because nobody has time, and the summary reads well.
In each case there is a person in the loop. In each case the loop has quietly opened.
Designing a review that holds
The research points at design, not at exhortation. These are the choices that make the difference.
Put the evidence beside the proposal. If checking a field means opening the original document and hunting for the line, it will not be checked. If the line is shown next to the field, checking takes a second and gets done. The cost of looking decides whether looking happens.
Send less to review. A person reviewing everything reviews nothing. Route the routine, high-confidence items straight through where the stakes allow it, and send the uncertain or consequential ones to a person. A short queue of items that deserve attention keeps attention.
Show uncertainty honestly. A proposal with no confidence shown is read as certain. Marking which fields the system was unsure of tells the reviewer where to look. This was one of the mitigating factors in the Goddard review.
Offer information before a verdict. A screen that says "approve?" invites a yes. A screen that says "the amount on the invoice is 1,240 and the amount on the purchase order is 1,420" invites a decision.
Keep the watch short. Vigilance falls within half an hour. Review works better in short sessions spread through the day than in one long block at the end of it.
Make one person accountable. In a follow-up study, Skitka and colleagues found that people who expected to have to justify their decisions made fewer automation bias errors. A review step with a name on it is checked more carefully than one owned by nobody.
Measure the reviewers, not only the system. Track how often a reviewer changes what the system proposed. A rate that falls to zero is not proof the system became perfect. Test the loop now and then with an item that is known to be wrong, and see whether it is caught.
We cover the screen itself in our guide to designing the approval step, and the routing in the page on approvals and routing.
The other side of the same problem
Automation bias is trusting a system more than it has earned. Its opposite, refusing to use a system that has earned trust, is as well documented and as costly. We write about that in Algorithm aversion: why people abandon AI after one mistake.
The target is the same in both cases: trust that matches how reliable the system really is. That is a property of the design. It does not arrive by itself.
References
- Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775 to 779. doi:10.1016/0005-1098(83)90046-8
- Goddard, K., Roudsari, A., and Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121 to 127. doi:10.1136/amiajnl-2011-000089
- Mackworth, N. H. (1948). The breakdown of vigilance during prolonged visual search. Quarterly Journal of Experimental Psychology, 1(1), 6 to 21. doi:10.1080/17470214808416738
- Mosier, K. L., and Skitka, L. J. (1996). Human decision makers and automated decision aids: made for each other? In R. Parasuraman and M. Mouloua (Eds.), Automation and Human Performance: Theory and Applications. Erlbaum. doi:10.1201/9781315137957
- Parasuraman, R., and Manzey, D. H. (2010). Complacency and bias in human use of automation: an attentional integration. Human Factors, 52(3), 381 to 410. doi:10.1177/0018720810376055
- Parasuraman, R., and Riley, V. (1997). Humans and automation: use, misuse, disuse, abuse. Human Factors, 39(2), 230 to 253. doi:10.1518/001872097778543886
- Skitka, L. J., Mosier, K. L., and Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991 to 1006. doi:10.1006/ijhc.1999.0252
- Skitka, L. J., Mosier, K. L., and Burdick, M. (2000). Accountability and automation bias. International Journal of Human-Computer Studies, 52(4), 701 to 717. doi:10.1006/ijhc.1999.0349
