Humans messing with AI

There is a tendency for human operators to trust themselves much more than the AI at hand, and this can frequently lead to suboptimal decisions

I’d written a version of this once, a few months ago, about my son Billy messing with our roomba Manja. I had written that while we talk a lot about “aligning AI with human values”, we don’t pay enough attention to humans that don’t let AI function as they should.

Driving this afternoon, my wife and I thought of another example of humans messing with AI - Bangalore’s traffic signals.

Now, Bangalore got a fancy set of traffic signals a year or two ago, that are “AI operated”. There are cameras monitoring vehicle flows in all directions, and the “AI” figures out based on that (and possibly some historical data) the optimal time to give green to each incoming direction.

When it started off, I remember it causing more pain than usual - and it had been attributed to the system calibrating itself, and doing so in terms of allocating green times to each lane randomly. Over a period of time, most of us here seem to have got used to it, and I don’t see too many complaints about it on social media.

Except - that the system is not always allowed to function. One thing we’ve observed (it’s anecdotal, and I typically don’t drive that much, and that too, don’t stray too far away from home) is that when we see long lines on any side of the signal, it is likely that the signal is being operated in manual mode (if you see “VAC” next to the lights, it means it’s the AI that is deciding the green time for each lane; if you see “MNL” it means there is a cop at the intersection who is overriding the AI).

Now sometimes there are valid reasons for overriding - mostly because of an incoming ambulance. And I’ve found that whenever there is a cop at a junction with a VAC signal, and they get aware of an inbound ambulance, they quickly override to manual mode so that the ambulance can be given right of way. However, I’ve also noticed that when this happens, a lot of them don’t immediately switch back to the tech - they try to handle the stuff manually for a while longer to try and stabilize the traffic post the disruption caused by the ambulance.

While AI is supposedly taking over the entire world right now, we see such examples all around us - where there is some AI that is supposed to do some job, but a human who is overriding it thinking that they know better. And I wonder why this happens. A few reasons off the top of my head:

• Trust: Humans don’t yet trust AI to make decisions. The AI may not make the optimal decision all of the time, and there is a selection bias where we tend to notice the inaccuracies more than the “usual”, and so we trust the AI less than we need to, and see the need to override.

• Desire for control: Handing over control (with the human having monitoring / overriding rights) means a loss of agency for the human. And when you are just observing the AI do its job, you inevitably want to overanalyze why it’s making certain decisions in certain ways, and then you decide you can do things by yourself. And so you apply the override even when you don’t need to

• Assuming you are better: In most tasks, most humans assume they are better than they actually are (Dunning-Kruger effect, etc.). And so, if you take an AI that you don’t fully understand, and hence don’t trust, and yourself, whose abilities you overestimate, it is easy to come to the conclusion that you can do a better job than the AI. And so you take control.

And this is something that I’ve seen myself do as well - because sometimes I fully don’t understand why the AI is doing or recommending something, I trust it less and imagine that had I been doing it manually I would’ve done a far better job.

What I don’t take into account at these points is this - I’m a human with finite (physical and mental) energy, and the AI has been built to assist me in my work. First of all, I can’t monitor it all the time, which means that there are decisions I won’t influence anyway. Second of all, with my energy being limited, it needs to be used only when absolutely necessary. Both of these mean that I better make use of the AI - even if, at the margin it is a worse decision maker, overall we will make far superior decisions.

As organizations begin to adopt more AI into their workflows, this is one kind of change they need to manage - to get the rank and file to let the AI do its work, which means a massive change in the way people normally operate. However, in the absence of this, AI impact will be limited.