There’s nothing new about products designed to be addictive. The food and drink industry spends millions making flavors that keep customers coming back. And smoking and alcohol are huge industries built around addictive products.
But what about technology? Designers of software, tools and apps obviously want to create “sticky” products that get us hooked on using them. But when does building a product that’s so good we can’t imagine living without it become dangerous to both our customers and our business?
Most pressingly, what are the implications of tech companies using AI that’s capable of learning about us and predicting our behavior to build products that are more addictive than ever before?
In March, shockwaves rocked the tech industry when a California court found that both YouTube and Meta services were intentionally designed in a way that could harm mental health.
$6 million in damages might seem like small change for a tech giant. But with the possibility of hundreds or thousands of similar cases following, that could scale rapidly. And unless regulators are able to more clearly define the boundary between using AI to drive engagement and using it to cause addiction, smaller companies building tools using algorithms come under fire, too.
I believe these developments shed light on what is perhaps the least well-understood dimension of responsible AI use: safety. As well as ensuring AI systems are trustworthy, secure, transparent and accountable, businesses and society at large still have a great deal to learn about the risks around mental health, addiction, and the unsafe paths that AI use can take us down.
From Engagement To Addiction
Modern digital products can be continuously optimized by AI to meet users’ needs, responding in real time to signals on what’s most likely to keep us scrolling or clicking.
A very straightforward example is the content recommendation engines that choose what will appear next in our social feeds. From the platform’s point of view, the most important questions an algorithm has to answer before deciding what to show us are around what’s most likely to keep us watching.
Today, most people are aware of the term “doom scrolling”. Why do people spend hours aimlessly browsing content when they’ve got better things to do? Because they’re being shown content that an AI, from all of the millions of pieces of content flooding the internet, believes will be most appealing to them.
This is very powerful, and when it crosses the line from a process that’s engaging to one that causes addiction (a word that signifies a mental health issue), it becomes a problem.
There are other aspects to the danger: just as we understand drug users can be pushed towards harder substances by softer gateway drugs, dynamic algorithms can drive users to more extreme or addictive content. This could include content by groups with extreme ideologies or pushing misinformation and propaganda.
And even if it doesn’t directly promote bad behavior, AI use has been shown to be capable of causing unhealthy behavioral patterns. Increasingly, we are seeing people become emotionally dependent on chatbots, blurring the lines between helpful interaction and unhealthy attachment.
For businesses, the risks are becoming clear. Allowing AI systems to shape customer behavior in negative ways could become a serious liability.
So, How Should Business Respond?
Regulators are already moving on this. Legislation like the EU Digital Services Act requires platforms and service operators to assess and mitigate systemic risks, which specifically includes the impact on mental health.
And the UK Online Safety Act states that harmful AI system design, as well as harmful content, must be considered when considering AI safety.
For businesses, the first step is understanding that this is a new category of risk. As well as data security and cybersecurity, the chance that your systems influence users in a harmful way has to be considered.
Most businesses won’t be ready to deal with this. But that won’t stop them from being responsible if their AI does cause damage in this way.
This could mean asking some tough questions. What are your AI systems really optimizing for? How do we spot when engagement tips over into addiction? And what warning signs or metrics should be used to trigger human intervention?
Most importantly, if challenged to do so, can you prove that you took all of this into consideration when assessing risks?
It’s a risk that will only become more present and real as more businesses adopt algorithmic methods to increase user engagement and retention. So now is the time to start figuring out how to respond to it.
