prefix cache packing — G = 4 rollouts
micro-batch
prompt (1,000 tok)
cached
resp₁
resp₂
resp₃
resp₄
4 × 100 tok
prompt processed
once
, not four times
3,000 tok saved
4,400
naive (before)
1,400
cached (after)