Agentic Serving：为什么推理基础设施要重新设计Agentic serving: why inference infrastructure has to be rebuilt
当一次请求变成几十轮对话、长上下文被反复复用、子 Agent 不断分叉，推理系统的优化目标就从“跑得快”变成“在真实交互里又快又省”。When one request becomes a forty-turn conversation, long context gets reused over and over and sub-agents keep forking, the optimization target shifts from raw throughput to fast and cheap inside a real interaction.
01 / 为什么问题变了
传统在线推理常被想象成一组彼此相对独立的请求：输入一段上下文，模型生成一段输出，任务结束。但 Agent 的运行方式不同。一个任务会持续多轮，每一轮都把历史、工具结果与新指令重新带入模型。上下文越来越长，而每轮新增的输出往往很短。
这意味着系统真正需要优化的，不只是单次吞吐，而是“长会话中的上下文复用、交互延迟与单位成本”。
因此，缓存不再只是性能技巧，而接近会话状态的一部分；调度不再只看队列长度，而要判断哪些请求拥有可复用前缀；prefill 和 decode 也需要更精细地拆分与配比。
01 / What actually changed
Conventional online inference is usually pictured as a stream of independent requests: feed in a context, the model produces an output, done. Agents do not work that way. One task runs for many turns, and every turn pulls history, tool results and new instructions back into the model. Context keeps growing while each turn adds only a short output.
What has to be optimized is not single-request throughput. It is context reuse across a long session, interactive latency, and cost per unit of work.
Caching stops being a performance trick and starts behaving like session state. Scheduling stops looking only at queue depth and starts asking which requests hold a reusable prefix. Prefill and decode need to be split and sized against each other instead of being treated as one stage.
02 / Agent workload 的形状
vLLM 团队在 AgentX 相关研究中强调了四个特征：长时间多轮会话、超长输入配短输出、高比例的 prefix reuse，以及大量子 Agent 分支。这样的 workload 会把 KV cache 容量、调度和跨实例复用推到系统设计中心。
02 / The shape of an agent workload
In their AgentX work, the vLLM team call out four properties: long multi-turn sessions, very long inputs paired with short outputs, a high prefix reuse ratio, and heavy sub-agent branching. That shape moves KV cache capacity, scheduling and cross-instance reuse to the center of system design.
拖动回合，观察上下文复用如何上升Drag the turn counter and watch context reuse climb
这个交互不是为了展示“炫技图表”，而是让读者直接理解结构变化：随着会话推进，绝大部分输入都变成已经见过的历史，只有少量新内容需要 prefill。文章中的图表应当服务认知，而不是装饰。
This is here so you can see the structural shift directly, not to show off a chart. As a session advances, most of the input becomes history the system has already seen, and only a thin slice needs prefill. A chart in an article should do cognitive work, or it should not be there.
03 / 优化变成三层协同
Data plane：让 cache 保持温热
第一层是 KV cache 的存储与复用。对于长会话，能否在 GPU、CPU、磁盘乃至分布式存储之间保留可复用状态，会直接决定下一轮是否要从头计算。
Execution plane：针对模型与 workload 选择并行方式
不同模型结构对 TP、DP、EP、CP 的收益不同。Agentic workload 又让长上下文和 decode 延迟变得更重要，因此“默认并行策略”越来越难成立。
Control plane：同时做调度与 P/D 配比
长 prefill 可能阻塞大量短交互回合。控制面要减少 head-of-line blocking，并让 prefill/decode 的容量更贴合当前请求组合。
03 / Optimization becomes three layers
Data plane: keep the cache warm
The first layer is where KV cache is stored and reused. For long sessions, whether reusable state survives across GPU, CPU, disk and distributed storage decides whether the next turn has to recompute from scratch.
Execution plane: choose parallelism for the model and the workload
Model architectures differ sharply in what they get from TP, DP, EP and CP. Agentic workloads add long context and decode latency to the list of constraints, so one default parallelism strategy no longer survives contact with production.
Control plane: schedule and size prefill/decode together
A long prefill can block a large batch of short interactive turns. The control plane has to cut head-of-line blocking and keep prefill and decode capacity matched to the request mix in front of it.
04 / 关键数据证据
下面不是为了做性能榜单，而是帮助建立量级感。公开文章里给出的 AgentX workload 与部署结果显示，Agent serving 已经足够特殊，值得成为独立优化对象。
04 / The numbers that matter
This is not a performance leaderboard. It is here to give you a sense of magnitude: the published AgentX workload and deployment figures make the case that agent serving is special enough to deserve its own optimization target.
05 / 对企业技术选型意味着什么
如果你的 Agent 只是偶尔触发一次模型并执行一个工具，这些优化可能不是首要矛盾。但当系统同时出现“长任务、多轮、多个 Agent、较高并发、强交互 SLA”时，推理层就不应再被当作黑盒 API。
Y 的建议是把四个问题加入架构评审：是否能观察 prefix cache 命中；长 prefill 是否影响短交互；扩容时是增加同构实例还是做 P/D 拆分；最终指标是否同时包含成本与 P90/P99 交互速度。
这也是 Y 文章页的设计原则：阅读过程中，目录、图表、数据证据和右侧 AI 小结保持联动，让读者可以从“结论 → 证据 → 结构 → 决策”来回跳转，而不是线性读完才知道重点。
05 / What it means for your stack
If your agent calls a model once in a while and runs one tool, none of this is your binding constraint. But once a system combines long tasks, many turns, multiple agents, real concurrency and a tight interactive SLA, the inference layer stops being a black-box API.
Y's recommendation: put four questions into architecture review. Can you observe prefix cache hits? Does a long prefill hurt short interactive turns? When you scale, are you adding more identical instances or splitting prefill from decode? And do your headline metrics include cost alongside P90/P99 interactive speed?
This is also the design rule behind Y's article pages: the table of contents, charts, evidence and the AI brief on the right stay linked while you read, so you can move between conclusion, evidence, structure and decision instead of reading straight through to find the point.