Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
The paper’s main gate is whether agent-control work can be batched before its launch deadline, and whether routing decisions stay on the GPU.
Using a pinned 851-session trace replay, the study reports that exact packing raises the feasible share from 30.19% under fixed windows to 43.00% at 100,000 target active sessions, K=256, and a 50 ms deadline. A separate mechanism test found that keeping a GPU-computed binary route decision on device beat returning four bytes to the host in all 36 tested configurations. The author also reports oracle-matching results across 14,557,440 batched invocations, while a fixed nested device graph that removed no host decision was slower in every tested configuration. The paper says a joined online runtime is still needed to measure achieved share, CPU displacement, and service-level benefit. ArXiv · AI/CL/LG's note
Using a pinned 851-session trace replay, the study reports that exact packing raises the feasible share from 30.19% under fixed windows to 43.00% at 100,000 target active sessions, K=256, and a 50 ms deadline. A separate mechanism test found that keeping a GPU-computed binary route decision on device beat returning four bytes to the host in all 36 tested configurations. The author also reports oracle-matching results across 14,557,440 batched invocations, while a fixed nested device graph that removed no host decision was slower in every tested configuration. The paper says a joined online runtime is still needed to measure achieved share, CPU displacement, and service-level benefit. ArXiv · AI/CL/LG's note
score 4