Will we ever trust artificial intelligence to act on its own? Probably not, at least for the foreseeable future.
One of the mandates within the European Union’s AI Act is a call for human oversight of artificial intelligence, though it’s not clear what forms this oversight will take. Expect similar legislation to take hold in other parts of the world. From a business standpoint, however, human oversight may be a practical necessity.
Expect many parties to have a role in overseeing AI. “Human oversight is a shared responsibility between AI developers and users,” wrote Dr. Johann Laux, postdoctoral fellow at the University of Oxford, in an analysis of the EU’s AI Act. “The obligation to execute oversight falls predominantly on users but should include developers for continuously learning AI systems on a case-by-case basis. Norms can be adjusted as to whether oversight needs to be constitutive or corrective of an AI’s output. So far, the emerging laws of oversight do not thoroughly distinguish these types of oversight.”
Of course, human oversight isn’t necessarily a panacea for error-free AI output. An analysis conducted two years ago concluded that “humans are mostly unable to perform their assigned oversight functions,” Laux also pointed out. Humans have been shown to both over-rely (automation bias) and under-rely (algorithm aversion) on algorithmic advice and fare badly at judging the accuracy of algorithmic predictions.” A case where human oversight is impractical is aviation systems, for example.
It’s a question of trust and confidence — both in AI systems and human judgement. One thing is clear, observers agree: we have plenty of confidence in AI for entertainment and small-scale recommendation engines — but it’s not ready to assume the larger tasks of business just yet on a manly autonomous basis.
“For personal entertainment on a small scale, we have likely built a certain level of confidence,” Ding Zhao, associate professor of mechanical engineering at Carnegie Mellon’s College of Engineering and head of the CMU Safe AI Laboratory. “However, for civil infrastructure such as self-driving, massive production, or healthcare, there are still issues that need to be addressed,” he said.
There are many instances now of when AI-driven decisions or processes were overruled or reversed by humans, Zhao added. “In the domain of autonomous vehicles, this is quite common,” he said. “In the healthcare domain, it is also a common practice that a human doctor would be able to overrule AI’s decisions.”
If anything, we’re a long way off from independent AI operations, especially when it comes to data transparency, said Carm Taglienti, chief technology officer and chief data officer at Insight Enterprises. “There are a lot of unknowns when it comes to the datasets used for training many large language models,” he states. “Not knowing where data is coming from erodes trust in an AI output and can also perpetuate biases.”
Trust in AI varies across industries as well, according to Yao Morin, chief technology officer at JLL and angel investor. “In industries such as finance and manufacturing where traditional AI has been used for a while, people have more trust in the predictive ability of AI.”
With other industries and applications, such as autonomous vehicles, the jury is still out, Morin cautioned.
Plus, modern AI approaches have become “less predictive and less replicable than classical AI models,” Taglienti warned. “Moving forward, there are some emerging trends that may improve the trust and determinism of generative AI models, such as action-based language models, causality models, and agent-based behavior.”
Still, demand for higher-order AI applications will keep increasing, Zhao predicts. “The rising cost of human labor due to an aging population and the decreasing cost of AI will drive us to use autonomous agents on a larger scale. We will be forced to solve the safety issues of AI within the next two decades.”
Humans need to be involved in AI decisions at these higher levels, and this requires thinking through responsibility levels. “The key is to clarify liability — who should be responsible for decisions, the user or the machine — the enterprise behind the technologies?” Zhao asked. However, he sees assigning authority or capability to overrule or reverse AI as “a political question rather than a scientific one.”
Ultimately, “it really comes down to whoever is being held accountable for the AI’s decisions,” Taglienti said. “It’s in their best interest – whether that be CIOs, CISOs, or IT leaders – to understand how an AI-driven decision was made and if there are any errors or mistakes. At the end of the day, there are real-world implications and risks to letting AI make decisions, and it needs to be used intentionally with clearly defined guardrails.”
AI can assist, but final decisions need to be human, said Taglienti. “After all, an AI agent today isn’t going to be able to grasp a lot of the nuances required for these inputs. Human overlay and verification remain crucial parts of AI-driven processes.”
Human intervention is crucial “when AI misses the mark or doesn’t meet expectations,” Taglienti said. “Maybe the output was inaccurate, too generic, or off-topic. That’s why testing and learning is so critical. It allows us to better understand the impacts and limitations of this technology and how we can use it safely and responsibly.”
The goal of a human-AI relationship “doesn’t have to be aiming for a lights-out, hands-off processes process,” Morin said. Instead, the key is “thinking about it in a balanced way, having human reasoning and intelligence as well as the machine’s vast knowledge base and capability of summarizing.”
