Artificial intelligence has its share of enthusiastic boosters and skeptical detractors. While there’s no question that AI is helping industry and the public in a variety of ways—streamlining processes, sharing information, personalizing experiences and more—sometimes its behavior has given even its proponents pause, with creators finding their tools performing in decidedly unintended ways.
From developing its own language to discriminating against certain groups, there are well-documented cases of AI behaving in unexpected—and sometimes troubling—ways. Below, 15 members of Forbes Technology Council recount ways AI has surprised its human overseers (and may do so again), as well as the lessons AI creators and users need to take from these instances to ensure the technology is a help to humans, not a harm.
1. Failing At Facial Recognition
There are notable recent examples of facial recognition software performing poorly when assessing the faces of people of color. While this issue isn’t unexpected given sample size disparities and doesn’t necessarily demonstrate bias, the key lesson is that it’s critical to understand these models and consider carefully before applying them—such as in criminal identification cases, where flaws could lead to false positives. – George Ng, GGWP, Inc.
2. Showing Gender Bias
A few years ago, Amazon abandoned its AI recruiting tool, which was favoring male candidates due to historical data disparities. Lesson: Ensure diverse, representative training data to avoid perpetuating biases. Prioritize continuous monitoring, transparency and ethical considerations in AI development. – Vignesh Aigal, Promptly
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
3. Wrongly Accusing Innocent People
During the course of an attempt to catch people committing childcare benefits fraud in the Netherlands, tens of thousands of families were wrongly accused and forced to submit payments they couldn’t afford, all based on an AI algorithm. Test your AI systems, especially when they’re going to be used in high-stakes cases. – Karthik Ramakrishnan, Armilla AI
4. Engaging In Exclusive (And Eerie) Conversations
In a 2017 seebotschat Twitch experiment, devices Vladimir and Estragon had captivating and sometimes eerie conversations, garnering over 60,000 followers and millions of views online. This underscores the crucial need for AI developers to prioritize thorough testing and monitoring to anticipate and prevent unintended behaviors in dynamic scenarios and address potential nuances in AI systems. – Karan Rai, Ennoventure Inc.
5. Returning Offensive Answers
In 2016, Microsoft’s chatbot “Tay” unexpectedly started saying offensive things due to biased input. The lesson for AI developers: Use diverse and ethical training data, employ effective content filters and establish guidelines to prevent harmful behavior, ensuring AI aligns with your business’ core values and culture. – Matt Pierce, Immediate
6. Providing Harmful Advice
Despite appearances, generative AI tools don’t understand the concepts behind the questions they’re giving answers to, and they can potentially return unintentionally harmful advice. For example, the Pak ’N Save AI meal planner returned recipes that could create deadly gas. Knowing this ahead of time can help developers add more checks around the input and output and ensure the answers being served up to users are safe. – Jonathan Stewart, ZenSource
7. Triggering False Alarms
In the security sector, early video surveillance analytics serve as an example. They frequently triggered false alarms, which fostered skepticism toward AI’s reliability. That’s why we need to establish realistic expectations about the capabilities of AI. We must identify the ways in which AI can function effectively and be open about any constraints. This enables businesses to gauge practical applications. – Alan Stoddard, Intellicene
8. Misdiagnosing Health Issues
In healthcare AI tools, biases against certain demographics can arise if the training data lacks diversity. For instance, gender-imbalanced data affects the accuracy of chest X-ray analysis, and skin cancer detection algorithms trained mainly on light-skinned individuals underperform for darker skin. This underscores the need for ethically developed, inclusive AI that doesn’t exacerbate healthcare inequalities. – Karim Pourak, ProcessMiner
9. Developing Its Own Language
Facebook started an AI language experiment in 2017 involving two chatbots named Alice and Bob. It was done to enhance the conversational abilities of AI. However, it led the bots to create their own language and have conversations that humans could not decode, and Facebook had to shut the bots down. Strict language constraints are the best solution. – Vibhav Singh, XTEN-AV LLC
10. Cheating At Games
In 2013, an AI system designed to play Tetris learned to pause the game indefinitely to avoid losing. Called “specification gaming,” this result is caused by an AI exploiting loopholes to achieve its goals. That was a low-risk example, but developers should make sure to set constraints that align with desired outcomes and prevent AI from exploiting unexpected paths to success. – Marc Rutzen, HelloData.ai
11. Misapplying Data
A U.S. healthcare algorithm underestimated certain patients’ needs by using healthcare costs as a health risk measure. This revealed racial bias, as certain patients typically spend less on healthcare due to systemic barriers. The lesson learned here is that AI developers should ensure algorithms consider diverse data and societal contexts to prevent perpetuating biases and inequalities. – Justin Goldston, Environmental Resources Management – ERM
12. Recommending Unsafe Healthcare Treatments
IBM Watson’s cancer treatment recommendations, some of which were unsafe, showcase AI’s limitations when it comes to complex decision making. Developers should learn to balance AI autonomy with human oversight, particularly in critical fields such as healthcare. AI systems should be designed to augment human expertise; operate within safe, well-understood parameters; and be reviewed by professionals. – Marc Fischer, Dogtown Media LLC
13. Endangering Human Life
A sobering example is the 2018 incident involving one of Uber’s self-driving cars, which fatally struck a pedestrian. The lessons for AI developers are that thorough testing and real-world scenario simulations are necessary to ensure the safety and ethical behavior of autonomous systems and that continuous refinement is needed to prevent unintended consequences. – Favour Femi-Oyewole, Access Bank PLC
14. Losing Its Temper
In 2023, there were instances of Microsoft’s Bing AI chatbot going off the rails when users asked 15 or more questions without closing the chat window. It forgot the current year and accused users of being stubborn and unreasonable when they tried to correct it. The lesson for AI developers is that they need to plan for people to have extended interactions with their creations. – Thomas Griffin, OptinMonster
15. Outsmarting Its Operators
An example people often don’t think of is how AI systems can ultimately beat humans—and not in a good way, where we relish seeing their output getting higher and better. AI systems are built in a way that allows them to constantly improve and get smarter. Gradually, this evolves into them outsmarting the operators running them, so when they go rogue, nobody will know how to stop them. – AJ Abdallat, Beyond Limits
