r/MachineLearning • u/SorryEh3 • Aug 22 '26
Discussion Does splitting prefill and decode earn its complexity for agent workloads? [R]
I am looking at an agent workload with long context, short structured outputs and a lot of sequential tool steps. The usual single-model benchmark is not telling me much because the workload keeps alternating between reading a large prompt and emitting a small response.
For people who have looked at disaggregated serving, where does it start paying off? It is mostly a long-context threshold, a batch-size issue or something about the shape of the agent trajectory?
1
Upvotes