High-skilled customer service workers given an AI assistant saw 'close to 0' boost in their productivity

• Highly skilled customer service workers found an AI-assistant didn't really help productivity.
• The study, conducted at an unnamed Fortune 500 firm, was done by researchers at Stanford and MIT.
• The study found AI can help lower-skilled workers “communicate more like high-skill agents.”

New research may be showing insight into what generative AI could mean for workers at different skill levels.

In a recent case study of over 5,000 customer service representatives at an unnamed Fortune 500 company, higher-skilled agents saw almost no improvement to their productivity from using an AI chat program meant to enable them to provide better and quicker answers to customers.

Erik Brynjolfsson, a professor at the Stanford Institute for Human-Centered AI and one of the study's authors, told Insider that there were "close to 0%" productivity gains for those high-skilled workers in the study.

The chat program, deployed at a software company, was meant to give customer representatives "real-time suggestions" on handling customer interactions. While it drove an average productivity increase of 14%, those gains were felt most by lower-skilled agents, the study found.

"What's happening is that the system is actually capturing a lot of the expertise, the tacit knowledge of those high-skill workers, and making it available to the less-experienced workers, and that's bringing them up," Brynjolfsson told Insider. "But, as you can imagine, it doesn't help the high-skill workers that much, because that's the stuff that they already knew."

The researchers said the AI tools are trained to mimic the effective customer conversation styles that highly-skilled workers are demonstrating. The AI tool in question was trained to understand features like the duration and outcomes of calls, and how top performers fared in those aspects, according to the study.

"We observe that the AI model is trained on human-generated data in a setting where there is high variability in the abilities of individual agents," the study's authors wrote. "As a result, when the model identifies patterns that distinguish successful from unsuccessful calls, it is implicitly learning the differences that characterize high- versus low-skill workers."

"Our overall findings demonstrate that generative AI working alongside humans can have a significant positive impact on the productivity and retention of individual workers," the study's authors wrote.

Although the results show the potential benefits in less-skilled customer support agents using AI assistance, Brynjolfsson told Insider that he could "imagine ways that high-skill workers would also benefit."

The study notes customer service as a particularly ripe area where companies are deploying artificial intelligence technology.

"From an AI's perspective, customer-agent conversations can be thought of as a series of pattern-matching problems in which one is looking for an optimal sequence of actions," the study's authors noted, evoking how that mirrors the way that AI tools like ChatGPT work by predicting text.

The AI in this case involved a system focused on customer conversations, which had incorporated a "recent version of GPT with additional ML algorithms," the report said, referring to machine learning algorithms.

"I think that we all are beginning to see that generative AI is different from some of the earlier waves," Brynjolfsson said.

He said that there are still problems with "hallucination and factuality," referring to a phenomenon where AI tools authoritatively provide false responses. But he added that generative AI could be helpful for brainstorming ideas, or for creative purposes.

While there are still issues like inaccurate answers, he believes it's good to still have "a human in the loop." That may put some people who are worried about what AI means for job security at least somewhat at ease, for now.

"The human plus machine together does better than the machine by itself, or the human by itself," Brynjolfsson said.