Legacy IT Infrastructure Isn’t Capable of Handling Agentic AI

This guest article was authored by Guilherme Soubihe, CEO of Latitude.sh

Infrastructure has quickly become one of the biggest bottlenecks for agentic AI success. A recent Harvard Business Study found that only 23% of business leaders have the infrastructure to support AI investment, and nearly half report that infrastructure costs are breaking budgets.

As agentic AI becomes the new enterprise standard, this gap will only continue to grow, unless root problems are addressed.

The cost problem predates agentic AI

For organizations looking to present a business case for investment in AI, unpredictable cloud costs create an undeniable obstacle. This problem stretches well past organizations that are simply running on outdated data center infrastructure. Cloud pricing models create ongoing cost pressure that continues to compound as usage increases.

Cloud compute spending on core workloads that have been running for years is eating up budgets. In fact, Flexera found that 29% of cloud spend around AI is wasted, which represents money that would otherwise be freed up for future AI investment. This isn’t a new situation, of course, but as AI workloads increase it is becoming more critical.

Enterprises are constantly making the tough choice between maintaining their existing infrastructure and investing in AI initiatives that support future goals. AI, specifically agentic solutions, have massive potential value for enterprises, cutting costs and producing huge strides in efficiency. The problem is, investment to get these initiatives off the ground can be massive. Organizations must first address the cost of the existing infrastructure before they can focus on the next generation of AI infrastructure. Variable egress charges, metered storage, and the overhead associated with virtualized environments create costs that are hard to forecast, and harder to control.

Fortunately, reevaluating cloud spending can free up budget and create opportunities to build out agentic solutions that generate massive value down the line.

Before looking ahead, leaders must evaluate their infrastructure to discern the processes that are essential, and what’s something that could be optimized or cut entirely. Robust audits are absolutely necessary if an organization wants to build the foundations of an agentic future.

Inference costs are the next big issue

Agentic AI is revealing a new cost structure that most IT and finance teams have not yet modeled. According to Gartner, AI inference costs per agentic workflow will increase more than fivefold through 2028. This is being driven by the so-called “inference paradox” where cost-per-token of running AI models continues to fall, while more-refined AI workflows consume a never-before-seen amount of tokens, causing total inference costs to skyrocket.

Basic chatbot interactions require very little compute. In contrast, calling on an agentic model for a task triggers a chain of planning, reasoning across multiple steps, and coordinating tool calls, which increases provider costs by a massive amount… and that’s at the very base. As workloads become more complex, the number of processes rises, meaning when enterprises make agentic solutions their default option, it will result in costs that are magnitudes higher than in optimized deployments.

For organizations managing AI on shared cloud infrastructure, costs appear as metered charges that are difficult to predict, and nearly impossible to cap without deliberate architecture decisions made ahead of scaling.

The ability to run inference is the true test for enterprise infrastructure readiness. Either the architecture is ready to take on the workload, or everything will grind to a halt.

Hardware migration isn’t easy

To solve this issue, some enterprises decide to migrate to efficiency-optimized accelerators to reduce compute costs and free up budget for agentic solutions. While seemingly productive on the surface, these migrations often cause more problems than they solve. The logic is straightforward: ARM-based architecture runs tasks with lower power draw, which in theory reduces the cost of running workloads at scale.

However, in practice, this means migrating x86 cloud instances to ARM-based cloud compute. This shift stays within the cloud but requires moving to a fundamentally different hardware architecture, requiring a lengthy process that ends in nearly no added net value when compared to simply addressing underlying cloud issues.

A migration is considerably more disruptive than previous cost projections account for. Moving an enterprise environment requires decoupling applications from hardware-specific assumptions, rebuilding dependencies, and re-testing the entire software stack in a new environment. Such migrations can take months, with associated costs often offsetting any major savings, at least within the visible planning horizon.

Freeing up cloud compute costs to invest in agentic AI is not achieved just by changing processor architecture. It’s achieved by removing inefficiencies within existing infrastructure, dissecting the layer below existing workloads, and renegotiating pricing models as opposed to the hardware.

Cost optimization is all about strategy

Gartner found that only 22% of organizations have been able to successfully scale AI across multiple business units. Infrastructure economics are one of the biggest bottlenecks stopping the remaining 78% from achieving success. Organizations that haven’t addressed the cost structure of their core cloud environment aren’t ready to absorb the inference costs of production agentic AI.

Planning from the ground up means accounting for what the compute bill will look like after scaling agentic AI. The organizations that make genuine progress on cloud costs are those that stop treating the problem as a procurement exercise, and instead approach it as a strategic one.

Procurement-driven cloud decisions optimize for unit price, focusing on cost per vCPU, per-gigabyte storage rate, and more. Strategy-driven decisions optimize for the total cost of running a given workload over time. That means starting from the very beginning: Selecting infrastructure partners that offer predictable pricing models so there’s no chance of surprise bills down the line.

Costs must be addressed now

The enterprises that will find agentic AI success are those who address the underlying cost structure of their cloud environment and implement cloud cost discipline before inference bills start skyrocketing. That entails auditing workloads to figure out what things actually cost to run, avoiding “easy” solutions like accelerator migration, and careful vetting of infrastructure partners.

Agentic AI can be extremely expensive, and those who can shoulder the burden of inference will reign supreme. The question is whether enterprises are prepared to shoulder those costs, or if they’ll sink under the weight.