Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
The paper says GPU-side agent control only pays when enough ready work exists and the route decision can stay on device.
It defines a “ready-cohort” bound for when small deterministic agent transitions expose enough parallelism for GPU execution. In one 851-session trace replay scaled to 100,000 active sessions, exact packing raised the feasible share from 30.19% under fixed partitions to 43.00%. A separate mechanism test found that keeping a GPU-computed binary decision on device beat returning four bytes to the host and redispatching across all 36 tested configurations. The author says a full online runtime is still needed to measure achieved share, CPU displacement, and service-level benefit. HF Daily Papers' note
It defines a “ready-cohort” bound for when small deterministic agent transitions expose enough parallelism for GPU execution. In one 851-session trace replay scaled to 100,000 active sessions, exact packing raised the feasible share from 30.19% under fixed partitions to 43.00%. A separate mechanism test found that keeping a GPU-computed binary decision on device beat returning four bytes to the host and redispatching across all 36 tested configurations. The author says a full online runtime is still needed to measure achieved share, CPU displacement, and service-level benefit. HF Daily Papers' note
score 4