Most enterprises have learned a hard lesson about infrastructure economics over the past decade: on-demand pricing is an expensive way to run a predictable workload.
Whether that baseline sits in their own data centres or in the public cloud, mature organisations now largely handle it the same way. They right-size it, commit to it through reserved instances, savings plans or owned hardware, and keep the expensive pay-as-you-go bursts for the spikes and the genuinely unpredictable.
The same discipline is now starting to arrive in enterprise AI, and the economics are, if anything, starker. Infrastructure, whether on-premises GPUs, reserved cloud capacity or a neocloud, can be clearly budgeted for. Tokens bought from frontier models typically cannot. An emerging pattern is to run AI workloads on committed infrastructure and use price-per-token frontier models only when the task demands.
Frontier models are the on-demand tier: powerful, elastic and priced by the unit.
“When you pay for infrastructure, it is much more predictable than paying per token” - Nutanix CTO
As AI increasingly moves into production, that unit, the token, is becoming one of the most unpredictable lines in the IT budget; or at least, one of the most predictably expensive ones – something that has given rise to a burgeoning “tokenomics” industry and a recognition that, from model choices to harness optimisation, getting AI right is going to take some refinement and a lot more robust oversight.
(Not least because developers will want to use the most high-powered models and the most expensive. To borrow a metaphor from Nutanix’s Dan Ciruli: “If you tell the developer you can drive the Rolls Royce all the time, they will.” Yet sometimes, of course, you don’t need to be driving the latest V12 “Phantom” to pick up the milk.)
“Earlier AI was an experiment. Now it’s going from experiment to scale,” as Nutanix CTO Manosiz “Mano” Bhattacharyya sums it up for The Stack. And without controls, bills can run into tens or even hundreds of millions of dollars, and they swing wildly.
An application that starts calling a reasoning model, or ingests badly cleaned data, can multiply token consumption overnight. Nutanix, which lets users manage VM-centric or containerized application infrastructure across on-prem, private, or public clouds, is now doing the same for AI workloads. The company recently pivoted its own developer AI workloads to open-weight models running on local GPUs. In doing so, it has cut costs by 80% and the GPUs will pay for themselves in 12 months, according to the company.
Control comes first
FinOps-style cost discipline works only because enterprises built controls around it: visibility into usage, policies on what runs where, and budgets that hold. AI bursting needs the same scaffolding, and it needs it before usage takes off, not after.
That’s the gap the Nutanix Agent Gateway is designed to fill. Now generally available in Nutanix Enterprise AI 2.7, it sits between an organization’s agents and the models and tools they call. Developers get a single API across hosted frontier models and self-hosted private ones, so changing where a request runs doesn’t mean rewriting the application. Platform teams get centralised visibility into token usage across model vendors, cost attribution by team and agent, and token-based rate limits and quotas enforced in one place. For agents that reach into business systems, governance for Model Context Protocol (MCP) servers adds tool-level access controls and a full audit trail.
Governance matters as much as cost. Bhattacharyya has watched the same pattern repeat with each new architecture; shadow IT became “shadow cloud-native” and now “shadow AI,” he warns wryly. A central AI gateway gives developers sanctioned, low-friction access, which removes much of the incentive to route around IT.
Most traffic stays home
Can most AI work run on committed infrastructure? It absolutely can, says Nutanix. Not least due to how agents work in practice. Bhattacharyya explains that an agentic loop typically starts with a planning step, in which the LLM works out what to do, followed by a chain of actions. Planning benefits from frontier-grade reasoning. Actions, which make up roughly 80% of token consumption, generally don’t.
“When it comes to actions, much simpler models can do a very good job.”
And as base model capabilities converge, Bhattacharyya argues, bringing AI closer to the data enterprise applications depend upon is another lever enterprises can pull in terms of latency and cost – whether they’re containerised or running on VMs.
Nutanix’s broader pitch is running VMs and Kubernetes side by side via Nutanix Cloud Clusters (NC2), with storage, networking and policy defined once across cloud or on-prem. “You don’t want separate operational dashboards,” says its CTO.
The same principle applies for generative AI workloads: Get oversight, governance, management in one place via its gateway – whether you’re running open-weight models on your own GPUs, making API calls to a frontier model on a hyperscaler, or a healthy mix of the two. Ultimately, enterprises that learned to stop paying on-demand rates for predictable workloads did so because they could see and control what they were running. AI deserves the same discipline: a predictable baseline on committed infrastructure, frontier models on demand, and a gateway in the middle.
Delivered in partnership with Nutanix.
