The Next Agent Cost War Is End-to-End Scheduling

AI cost discussions begin with the price per million tokens. A paper released on August 5, drawing on a Microsoft Azure production study and controlled work with open-source frameworks, shows why that measure is incomplete. An agent is not one inference request. It is a sequence of model calls, tool execution, orchestration decisions, waiting, and recovery.

Conventional online inference is relatively regular: a request reaches a GPU, the model generates, and a response returns. Agent execution is fragmented. A model plans, the CPU runs orchestration, a tool touches a file or network service, and the result returns to a model—possibly a different one. Subtasks branch and rejoin, creating sudden shifts between compute resources.

One important observation is that the CPU returns to the critical path. Tool runtimes and orchestration live on the host. CPU load may look low on average and then spike when tools complete or parallel branches merge. Provisioning for the average creates congestion; provisioning for every peak strands capacity.

GPU economics change too. Agents use different models, context lengths, and execution patterns. State can occupy memory while an agent waits for a tool, then many agents can resume together. Uniform servers and simple request queues are a poor match for heterogeneous, bursty workflows.

The paper's Agora prototype pools CPU resources by role, harvests idle cores, protects tail latency during tool spikes, and uses memory oversubscription and prefetching to place more agents on GPUs. The exact gains need broader validation, but the architectural direction is clear: infrastructure must schedule a task graph, not just an isolated model request.

That changes unit economics. A product can cut model prices by 30 percent and still spend more if a fragile workflow retries three times. A more expensive model may be cheaper per accepted result if it plans reliably, makes fewer tool calls, and recovers locally. The useful metric is cost per successful task.

Instrumentation should therefore include CPU time, GPU wait, tool latency, retries, parallel fan-out, context reconstruction, human takeovers, and the sunk cost of failed work. Without task-level attribution, a team sees a rising cloud bill but cannot distinguish an expensive model from a slow tool or runaway orchestration.

Parallelism and caching need the same task-level view. Ten agents exploring the same vague problem often create duplicated cost rather than speed. Parallel branches should be independent, comparable, budgeted, and easy to stop. Caches should preserve verified intermediate artifacts and checkpoints, not only prompt prefixes that become stale when task state changes.

Recovery is another major cost center. If a process restart or expired credential forces a long task back to step one, the system repeats model and tool work. Explicit nodes with inputs, outputs, and validation state allow execution to resume from the last trustworthy checkpoint and make bottlenecks visible.

Teams do not need custom datacenter software to act on the finding. Give every task an ID and budget, separate model time from tool and queue time, save recoverable checkpoints, and calculate cost against accepted outcomes. Model prices will keep falling, but stronger agents will also take on longer work. The durable advantage will come from scheduling the complete execution path well.