Little by little, trust in AI output — in both its operational and generative forms — is leading to lights-out, hands-off processes on an increasingly wider scale. But how much authority will humans have — and should have — to step in and overrule AI decisions?
For example, a potentially valuable bank customer may be denied a loan by an AI system. Or, an AI-based recruitment system may deliver biased or sexist results.
However, worrying about human intervention to stop AI-driven processes may mean it’s already too late. Things shouldn’t even reach this stage, said James Hendler, professor at Rensselaer Polytechnic Institute and director of RPI’s Future of Computing Institute.
“If systems using AI are correctly designed to be used interactively, then overruling or reversing becomes the wrong way to look at things,” said Hendler. “Together the systems should be solving the problems, with both human and technological interaction – especially in cases where human expertise is needed.”
This is where thoughtful “responsible-by-design” principles may help ensure balanced human interaction with AI systems, said Sunil Senan, global head of data, analytics and AI for Infosys. “The ease of overriding AI should be a carefully considered design decision based on the specific application and its level of risk, transparency, user expertise, and the evolving landscape of AI development.”
Senan pointed to examples of situations where humans have had to intervene to prevent AI-driven decisions. “From sifting through flagged transactions to identify false positives in fraud detection tools, to providing a safety override in self-driving cars, and making critical judgments on sensitive social media content, humans ensure ethical considerations, critical thinking, and handling situations beyond the AI’s training.”
It depends a lot, of course, on the criticality of the use case. The degree of human oversight depends on “the specific type of AI and our tolerance for risk in particular domains,” said Timothy Harfield, head of product marketing for ORO Labs. “In marketing, for example, machine learning has proven very effective for selecting the right offer to present to a particular person because it performs better than simple rule-based approaches. If it fails, the relative business risk is low.”
Areas of higher risk may include financial transactions, transportation, and even certain kinds of content creation, Harfield continued. “Here, “it’s vital that humans provide oversight and have the ability to exercise practical judgment under exceptional conditions.”
Lower-risk operational AI tasks may include “beauty apps personalizing video recommendations and chat interactions, which can be hands-off,” Senan said. “Similarly, mobile brands leverage AI and machine learning to forecast returns and optimize warranty provisioning. However, for tasks with higher risk or evolving regulations, such as creative content generation —generative AI — human oversight remains crucial.”
In high-risk AI systems, “businesses need to insist on ease of review and override,” Senan continued. “In fact, in many AI systems the output is defined as a first draft, so in financial report creation or an ad creation with text and image, what is created is expected to be only a base version with humans then completing it.”
Whereas in low-risk AI systems “such as chatbots or product review summarization on websites, constant human review is not only less necessary but also infeasible,” he added.
With humans and AI working together, “humans provide domain expertise and critical thinking, while AI handles data analysis and pattern recognition,” said Senan. “In all human in the loop systems, humans very often overrule AI decisions in the initial phases. Over time two things happen, AI learns and gets better while the human gets more familiar with the system. More explainability built into AI also causes the overruling of AI decisions by humans to go down.”
Ultimately, explainability is the key, and “the introduction of AI into a process should take the form of recommendations accompanied by a level of transparency with respect to how decisions are derived,” said Harfield. “This way, decision-making is accelerated, and trust is developed in the system in advance of full automation, should that be the right approach.”
The bottom line is that trust in unattended AI for higher-level decisions mainly applies to narrow scenarios at this point, and will likely remain so for some time to come. “There may be some niches where AI systems can handle tasks autonomously, but these are generally small niches — such as flying a drone between two pre-defined points — and not the sort of tasks most people think about when talking about modern AI,” said Rensselaer’s Hendler. Overall, he added, “I do not believe there is enough capability — and therefore trust — in AI for hands-off processes, and I don’t see that changing any time soon.”
