Lessons learned from agentic AI leaders reveal critical deployment strategies for enterprises

Companies precipitate AI agents in production – and many of them will fail. But the reason has nothing to do with their AI models.

The second day of VB Transform 2025Industry leaders have shared lessons hardly won by the deployment of large -scale AI agents. A moderate panel by Joanne Chen, general partner at Capital of the foundation, Included Shawn Malhotra, CTO Rocketwhich uses agents through the property trip of mortgage subscription to customer chat; Shailesh Nalawadi, product manager at Birdwho builds agency customer service experiences for companies in several verticals; and Thys Waanders, please transformation from AI to Cognilewhose platform automates customer experiences for major corporate contact centers.

Their shared discovery: companies that build first assessment and orchestration infrastructure are successful, while those that rushed to production with powerful models failed on a large scale.

>>See all of our Transform 2025 coverage here<

The reality of the king: beyond the simple cost cutting

A key element of engineering You have an agent Because success includes the return on investment (king). Early IA agent deployments have focused on reducing costs. Although this remains a key component, business leaders now report more complex return on investment models that require different technical architectures.

Cost reduction wins

Malhotra shared the most dramatic cost of rocket companies. “We had an engineer [who] In about two days of work, he was able to build a simple agent to manage a very niche problem called “transfer tax calculations” in the process of the process. And that two days of effort allowed us one million dollars a year in costs, “he said.

For Cognpy, Waanders noted that the cost per call is a key metric. He said that if AI agents are used to automate parts of these calls, it is possible to reduce the average handling time per call.

Income generation methods

Saving is one thing; Getting more income is another. Malhotra reported that his team had seen conversion improvements: while customers get the answers to their questions faster and have good experience, they convert to higher rates.

Proactive income opportunities

Nalawadi has entirely stressed new income capacities thanks to proactive awareness. His team allows a proactive customer service, reaching out before the customers realize that they have a problem.

An example of food delivery illustrates it perfectly. “They already know when an order is late, and rather than waiting for the customer to get angry and call them, they realize that there was an opportunity to get ahead of it,” he said.

Why the agents of the AI ​​break in production

Although there are solid return on investment opportunities for companies that deploy an agency AI, there are also challenges in production deployments.

Nalawadi has identified the basic technical failure: companies build AI agents without evaluation infrastructure.

“Before you even start building it, you should have an assessment infrastructure,” said Nalawadi. “We were all software engineers. No one unfolds in production without performing unit tests. And I think that a very simplistic way to think of Eval is that it is the unit test of your AI agent system. ”

Traditional software test approaches do not work for AI agents. He noted that it is simply not possible to predict all possible entries or write complete test cases for natural language interactions. The Nalawadi team has learned this through customer service deployments through retail services, food delivery and financial services. Standard quality insurance is approaching missed edge cases that have emerged in production.

AI AI test: the new quality insurance paradigm

Given the complexity of AI tests, what should organizations do? Waanders solved the test problem by simulation.

“We have a feature that we are publishing soon which is to simulate potential conversations,” said Waanders. “They are therefore essentially AI agents who test the AI ​​agents.”

The tests are not only the quality of conversation tests, it is a large -scale behavioral analysis. Can he help understand how an agent reacts to angry customers? How does it manage several languages? What happens when customers use slang?

“The biggest challenge is that you don’t know what you don’t know,” Waanders said. “How does it react to everything that anyone could find? You only discover it by simulating conversations, really pushing it in thousands of different scenarios.”

The approach tests demographic variations, emotional states and on -board cases that human QA teams cannot cover in depth.

The explosion of complexity to come

The AI ​​agents AI manages unique tasks independently. Business leaders must prepare for a different reality: hundreds of agents by organization learning from each other.

The implications of infrastructure is massive. When the agents share data and collaborate, the failure methods are multiplying exponentially. Traditional surveillance systems cannot follow these interactions.

Companies must now architect this complexity. The modernization of infrastructure for multi-agent systems costs much more than building it properly from the start.

“If you are going quickly in what is theoretically possible, there could be hundreds in an organization, and maybe they learn from each other,” said Chen. “The number of things that could happen explode. The complexity explodes. “