Agentic serving: why inference infrastructure has to be rebuilt

When one request becomes a forty-turn conversation, long context gets reused over and over and sub-agents keep forking, the optimization target shifts from raw throughput to fast and cheap inside a real interaction.

01 / What actually changed

Conventional online inference is usually pictured as a stream of independent requests: feed in a context, the model produces an output, done. Agents do not work that way. One task runs for many turns, and every turn pulls history, tool results and new instructions back into the model. Context keeps growing while each turn adds only a short output.

What has to be optimized is not single-request throughput. It is context reuse across a long session, interactive latency, and cost per unit of work.

Caching stops being a performance trick and starts behaving like session state. Scheduling stops looking only at queue depth and starts asking which requests hold a reusable prefix. Prefill and decode need to be split and sized against each other instead of being treated as one stage.

02 / The shape of an agent workload

In their AgentX work, the vLLM team call out four properties: long multi-turn sessions, very long inputs paired with short outputs, a high prefix reuse ratio, and heavy sub-agent branching. That shape moves KV cache capacity, scheduling and cross-instance reuse to the center of system design.

This is here so you can see the structural shift directly, not to show off a chart. As a session advances, most of the input becomes history the system has already seen, and only a thin slice needs prefill. A chart in an article should do cognitive work, or it should not be there.

03 / Optimization becomes three layers

Data plane: keep the cache warm

The first layer is where KV cache is stored and reused. For long sessions, whether reusable state survives across GPU, CPU, disk and distributed storage decides whether the next turn has to recompute from scratch.

Execution plane: choose parallelism for the model and the workload

Model architectures differ sharply in what they get from TP, DP, EP and CP. Agentic workloads add long context and decode latency to the list of constraints, so one default parallelism strategy no longer survives contact with production.

Control plane: schedule and size prefill/decode together

A long prefill can block a large batch of short interactive turns. The control plane has to cut head-of-line blocking and keep prefill and decode capacity matched to the request mix in front of it.

04 / The numbers that matter

This is not a performance leaderboard. It is here to give you a sense of magnitude: the published AgentX workload and deployment figures make the case that agent serving is special enough to deserve its own optimization target.

05 / What it means for your stack

If your agent calls a model once in a while and runs one tool, none of this is your binding constraint. But once a system combines long tasks, many turns, multiple agents, real concurrency and a tight interactive SLA, the inference layer stops being a black-box API.

Y's recommendation: put four questions into architecture review. Can you observe prefix cache hits? Does a long prefill hurt short interactive turns? When you scale, are you adding more identical instances or splitting prefill from decode? And do your headline metrics include cost alongside P90/P99 interactive speed?

This is also the design rule behind Y's article pages: the table of contents, charts, evidence and the AI brief on the right stay linked while you read, so you can move between conclusion, evidence, structure and decision instead of reading straight through to find the point.