Most people who have introduced new software to a team have seen a version of this. The tool arrives. It works. For a few weeks it does a tedious job faster and more accurately than the job was done before. Then it makes one mistake that everyone sees.
A week later, half the team is back to doing the work by hand.
Nobody decided the tool was worse than the old way. On the numbers it was better. Something else happened, and psychologists have a name for it.
The finding
In 2015 Berkeley Dietvorst, Joseph Simmons and Cade Massey ran a set of forecasting experiments. Participants had to predict things such as how well applicants would do in a business school programme. They could rely on their own forecast or on a statistical model. Some of them first got to watch the model perform, including its misses.
The model was clearly better than the people. Participants who watched it saw that. They were still less likely to choose it than participants who had never seen it work at all. Seeing it err, even while seeing it win, reduced their willingness to use it.
The researchers called this algorithm aversion. The key detail is the asymmetry. Participants who watched a human forecaster make the same kind of error did not lose confidence in the same way. A person's mistake is read as a slip. A machine's mistake is read as proof that it is broken.
An old argument with new stakes
The background to this goes back to 1954, when the psychologist Paul Meehl published a short book comparing two ways of making predictions about people: expert judgement, and simple statistical rules. The rules did as well or better in nearly every comparison he could find. Meehl later called it his disturbing little book.
The result has held. A meta-analysis led by William Grove in 2000 gathered 136 studies of predictions about health and behaviour. Mechanical prediction equalled or beat clinical judgement in the large majority of them. Robyn Dawes showed in 1979 that even crude linear models, with equal or randomly chosen weights, tended to beat expert judgement.
And yet the rules were rarely adopted. The evidence that a method works has never been enough to get people to use it. That gap is where algorithm aversion lives.
The opposite finding is also true
It would be wrong to conclude that people simply distrust machines.
In 2019 Jennifer Logg, Julia Minson and Don Moore reported a series of experiments in which people received the same piece of advice, labelled as coming from an algorithm or from a person. People gave more weight to the advice when they believed it came from an algorithm. The authors called it algorithm appreciation.
The two results fit together once the conditions are laid side by side.
- People start out willing to trust an algorithm, often more than a stranger.
- That willingness drops sharply once they watch it make a mistake.
- It is weaker when the choice is between the algorithm and their own judgement, as opposed to someone else's.
- It is weaker among experts. In the same study, professionals who made forecasts for a living discounted the algorithm's advice, and were less accurate for it.
So the pattern is not rejection. It is brittle trust: high at first, and unforgiving.
The kind of task matters
Not all tasks draw the same resistance.
Noah Castelo, Maarten Bos and Donald Lehmann found in 2019 that people trust algorithms less for tasks that seem subjective, such as predicting how funny a joke is or recommending a romantic partner, than for tasks that seem objective. The same task drew more trust when it was described in terms of what could be measured.
Chiara Longoni, Andrea Bonezzi and Carey Morewedge studied reluctance to accept medical advice from an automated provider. The cause they identified was a belief that a machine cannot take a person's individual circumstances into account. They called it uniqueness neglect: the worry that an algorithm treats everyone as average, and I am not average.
Both findings point the same way. Resistance is highest where people feel the work involves judgement about particulars, which is how most professionals see their own work.
Too much trust and too little are one problem
John Lee and Katrina See framed the issue in 2004 in a paper that is still the standard reference. The goal, they argued, is not more trust. It is calibrated trust: trust that matches how reliable the system really is.
Trust that runs ahead of reliability leads to misuse, where people stop checking. We cover that in Automation bias: why people stop checking the machine. Trust that falls behind reliability leads to disuse, where a working system is ignored. Algorithm aversion is the second failure.
A team can show both in the same month. They approve every draft without reading it until the day one goes wrong, and then stop using the system at all.
What helps
Give people a way to adjust the output. In 2018 Dietvorst, Simmons and Massey tested a simple change. Participants chose between their own forecasts and the model's. Some were told that if they chose the model, they could adjust its forecasts. Willingness to use the model rose considerably. It rose even when the permitted change was small. People did not need much control. They needed some. A system that produces a draft for a person to edit is using this result. A system that sends on its own is ignoring it.
Compare it with the present, not with perfection. An error rate on its own reads as failure. The same error rate beside the error rate of the current process reads as an improvement. Running a system alongside the existing way of working before launch, and showing both sets of results, gives people the comparison they need. Our guide to shadow mode before launch describes how.
Plan the first mistake. The first visible error is coming. What it means to the team depends on whether they were told to expect it. Before launch, say what kinds of mistake the system is likely to make, how each is caught and what happens next. An error that was predicted confirms that the people in charge understand the system. An error that surprises everyone suggests they do not.
Start where the work is seen as objective. Copying figures from an invoice or finding an open slot in a calendar will meet less resistance than drafting advice. Early wins in plain tasks build the record that harder ones will need. This is part of how we suggest choosing the first workflow.
Show the evidence. When a result comes with the line of the document it was taken from, a wrong result has a visible cause: it read the wrong line. That is a specific, fixable fault. A wrong result with no explanation is evidence of general unreliability, and it is judged that way.
Keep the expert as the author. People accept help with their work more readily than a replacement for their judgement. A system that prepares and a professional who decides answers the worry about being treated as average, because the person who knows the particulars still has the last word.
The larger point
A system is not adopted because it is accurate. It is adopted because the people using it hold a level of trust that survives its first error. That is designed in before launch or it is not there when it is needed.
References
- Castelo, N., Bos, M. W., and Lehmann, D. R. (2019). Task-dependent algorithm aversion. Journal of Marketing Research, 56(5), 809 to 825. doi:10.1177/0022243719851788
- Dawes, R. M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34(7), 571 to 582. doi:10.1037/0003-066X.34.7.571
- Dietvorst, B. J., Simmons, J. P., and Massey, C. (2015). Algorithm aversion: people erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114 to 126. doi:10.1037/xge0000033
- Dietvorst, B. J., Simmons, J. P., and Massey, C. (2018). Overcoming algorithm aversion: people will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155 to 1170. doi:10.1287/mnsc.2016.2643
- Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., and Nelson, C. (2000). Clinical versus mechanical prediction: a meta-analysis. Psychological Assessment, 12(1), 19 to 30. doi:10.1037/1040-3590.12.1.19
- Lee, J. D., and See, K. A. (2004). Trust in automation: designing for appropriate reliance. Human Factors, 46(1), 50 to 80. doi:10.1518/hfes.46.1.50_30392
- Logg, J. M., Minson, J. A., and Moore, D. A. (2019). Algorithm appreciation: people prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90 to 103. doi:10.1016/j.obhdp.2018.12.005
- Longoni, C., Bonezzi, A., and Morewedge, C. K. (2019). Resistance to medical artificial intelligence. Journal of Consumer Research, 46(4), 629 to 650. doi:10.1093/jcr/ucz013
- Meehl, P. E. (1954). Clinical versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. University of Minnesota Press. doi:10.1037/11281-000
