Does AI Actually Increase Productivity? These Ivy League Researchers Came to a Surprising Conclusion.

It’s taken as a given: AI boosts your productivity. But…does it really?

A group of researchers at the business schools of Harvard, MIT and Warwick universities along with the University of Pennsylvania’s Wharton School wanted to find out. So they assembled a team of 758 Boston Consulting Group employees, and put them through a few experiments to study genAI’s effect on their work.

Related: 7 Ways AI Made My Work Smarter — and Not Harder

The research

The investigators separated the employees, all consultants, into two groups for different experiments. In both, the subjects were first asked to solve a problem without using AI to establish a performance baseline. Then their group was given a challenge where some of the consultants were able to use GPT-4, while others were not.

In the first group, the assignment was to come up with footwear ideas for niche markets.

For the second group, the researchers purposely designed a task where the AI would make a mistake in its answer. In this challenge, the employees had to solve a problem for a company by assessing interviews with its insiders and evaluating quantitative data. Would the consultants carefully check the AI’s work?

At the end, researchers measured how productive everybody was.

The results

In the first group (who had to brainstorm footwear ideas), generative AI helped a lot. On average, the consultants using it were able to finish 12.2% more tasks and do them 25.1% faster than their peers who didn’t have access. The quality of their work was also 40% higher. Interestingly, generative AI proved to be a skill-leveler: It raised everyone’s performance — but less-experienced consultants surged 43% compared to those in the top tier (who only improved 17%).

For the second group tackling the company problem, not surprisingly, the outcome was very different: AI worsened their performance. The consultants who used GPT-4 reached correct answers only 60% to 70% of the time. But those who went without — relying just on their human brains — were correct about 84.5% of the time.

Related: Don’t Stand in the Way of AI — How Artificial Intelligence Can Turn Us Into Better Leaders and Entrepreneurs

What we’ve learned

We already know that AI is good at helping us brainstorm, and that it’s not great at making critical judgments about complex information — which partially explains the results of these two very different tasks. But the researchers gained further insight by watching how people were using GPT-4 in both.

The lesson is this: Be careful. Although generative AI significantly improves productivity for tasks it’s good at, it can throw you off for things it’s not. When the consultants used AI to do more complex tasks, they tended to blindly accept its outputs without seriously interrogating it — the work version of being asleep at the wheel.

How to use this

“The key to genAI is to try it for everything, to get a sense of what it’s good and bad at,” says study investigator Ethan Mollick, a Wharton professor and author of the new book Co-Intelligence.

To use it more effectively, he suggests treating it like a human being. “The best prompters I know have no engineering skills, but they’re good at working with people,” he says. Even better, tell the AI what kind of person to be. It’s designed to have a generic default personality but can adapt however you need. For example, ask it to act as a data geek or witty comedian. The first would be better at analyzing a business question; the second at writing ads that make people crack up.

Related: AI Is My Meal Planner, Editor, Assistant and More as a Working Mom — Here’s How You Can Use AI to Conquer Multitasking, Too.