CIOs not entirely sold on generative AI copilots

Microsoft and other vendors are touting the productivity gains their enterprise AI assistants can help achieve. Not all IT leaders are convinced the return on investment is there yet.

A new breed of AI assistant has set its sights on the enterprise user in recent months, with Microsoft and other vendors promising huge productivity gains that offset the cost.

But Microsoft still has work to do on its value proposition. Its Copilot for Microsoft 365, a high-profile offering among the growing list of AI agents, costs $30 per seat per month, with a 300-seat minimum. The enterprise version of Gemini, Google’s AI assistant, is similarly priced.

Price is a big reason why the University of California, Riverside, is hedging its bets and evaluating several AI tools, says Matthew Gunkel, the university’s CIO. AI technology is moving so fast, and the ROI is still murky, making it difficult to commit to one AI assistant, he says.

“We’re still a ways away from clearly understanding who might be winners and losers in this space,” he adds.

The benefit of an AI assistant is evident for some jobs, such as programmers, he says. AI tools can help coders clean up logic and coding errors and find security problems, and they may also help to accelerate programmers’ skills, cutting the sunk cost of internal training, he suggests.

Gunkel is leaning toward offering a couple of AI assistant options, including Microsoft Copilot, to employees later this year. However, he doesn’t yet see a clear case for general-purpose office AI assistants like Copilot. “Right now, am I inclined to pay at scale? No,” he says. “But am I inclined to pay for small portions of the population? Yeah, that makes sense.”

Considering the benefits

Copilot, and similar AI agents, can generate entire documents from short prompts, create slide presentations, automate repetitive tasks, pull together information from multiple sources, and summarize emails, chats, and meetings for employees flooded with all three, Microsoft says. Copilot can automatically assign tasks when an employee promises to do something in a meeting or an email.

Microsoft also offers Copilot for specific roles, like sales and security, with Copilot for Security summarizing incidents, conducting impact analysis, and providing guided response to incidents.

Microsoft has been touting the value of Copilot, saying the AI assistant makes 70% of users more productive. More than three-quarters of those answering a survey from Microsoft say they don’t want to give Copilot up once they’ve used it, and 85% said the AI tool helps them produce a first draft faster.

Three quarters of those surveyed say Copilot saves them time by helping to search their files, according to a November report from Microsoft.

“As use of generative AI at work spreads, the real opportunity is to not only transform personal productivity but lift the capability of the entire organization,” the report says. “Copilot sets a new baseline — one where every employee gains the skills to write, design, code, analyze data, and more. And it amplifies expertise, taking work from good to exceptional.”

The fear of the blank page

Many of those surveyed by Microsoft aren’t paying the bill, however. While some CIOs and other tech leaders believe in the future of AI assistants, the functionality isn’t quite there yet, some say.

Mike Mason, chief AI officer at IT consulting firm Thoughtworks, has arrived at a similar wait-and-see spot as Gunkel has, even though Mason is a self-described “AI optimist.”

Thoughtworks has been an early tester for Google Gemini, and the AI assistant has been particularly helpful to employees who speak English as a second language, Mason says.

Gemini has also helped employees with writing prompts, he adds. “You’re helping people avoid the curse of the blank page,” Mason says.

Thoughtworks also uses Github Copilot, an AI coding assistant not affiliated with the Microsoft product of the same name.

“For programmers, it seems to be a lot easier to justify,” he says. “When you can get Copilot to spit out a few blocks of code, even if you’re even saving yourself five to 10 minutes a day, $30 a month becomes very reasonable, given what developer time costs.”

But the cost/benefit analysis of an office AI assistant is less clear, Mason says. “The productivity is there, but I can also see a lot of organizations multiplying the number of employees by $30 and saying, ‘Well, that’s an extra cost that I don’t want to take on,’” he adds.

It’s telling that Thoughtworks, an early tester of Gemini, still hasn’t decided how broadly it will roll it out.

“It’s funny because on the one hand, $30 a month doesn’t seem very much if you can give your people a productivity boost,” he says. “But on the other hand, especially in today’s climate, everybody’s looking very, very squarely at all costs.”

The value is coming

Still, enterprise AI assistants will soon create “immense value” for enterprises, says Sunny Bedi, chief information and data officer at Snowflake, a cloud computing provider that has developed its own AI assistant for generating SQL using natural language.

“Eventually, all employees will have their own, customized AI assistants that boost productivity in their roles,” Bedi says. “Whether it’s personalizing the new hire onboarding experience or automating routine tasks like setting up calendars and email responders, AI assistants have the potential to make all of our lives easier.”

Another believer in AI assistants is Linda Simovic, who helped build Copilot for Sales when she served as chief product officer for customer engagement applications at Microsoft between late 2021 and late 2023.

Simovic found Copilot useful as a Microsoft employee. “The whole company is dog-fooding the product,” says Simovic, now executive vice president and chief product officer at People.ai, which builds specialized AI tools. “I really loved the email summarization because I can’t stand when people write me long emails.”

Simovic also appreciated the Copilot feature that provided summaries of meetings when she joined late, saving her from pinging other participants.

But like UC Riverside’s Gunkel, she sees the AI product marketplace as being in flux. “The interesting thing about AI is that they’re going to build a lot of stuff, not knowing what’s going to stick because it’s a new frontier,” she says. “We were just trying to experiment and get people to try it.”

The ROI of AI assistants is also difficult to measure for many jobs, she adds. If Copilot saves an employee 10 minutes by summarizing the 150 emails in his inbox, then calculating the ROI depends on what the employee does with the time saved, she notes. If he then spends that time on Facebook, there’s no return on investment.

But when AI assistants are tailored for specific employee roles, the productivity increase can be huge, Simovic says. For some employees, an AI assistant can drive 20% to 40% productivity gains, she says.

AI assistants will continue to evolve, with Microsoft and other companies offering more targeted experiences based on employee roles, Simovic adds. But the price of these AI assistants may turn some companies away from widespread rollouts in the short term.

“What Microsoft built is a foundation,” she says. “But at that price point, it’s pretty hard to swallow. If you have thousands of employees, $30 per user per month does add up.”