How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF
The paper finds reward-model scoring speed depends less on C++ itself than on ONNX Runtime and batching choices.
In CPU tests, the authors’ native C++ engine beat PyTorch eager mode, an HTTP baseline, and FastAPI with non-overlapping confidence intervals. On GPU, it still beat PyTorch and FastAPI, but the HTTP baseline was faster. Correctness checks matched the PyTorch reference within small tolerances before benchmarking. The authors argue faster scoring mainly frees shared compute for rollout generation, rather than automatically shortening the whole RLHF step. HF Daily Papers' note
In CPU tests, the authors’ native C++ engine beat PyTorch eager mode, an HTTP baseline, and FastAPI with non-overlapping confidence intervals. On GPU, it still beat PyTorch and FastAPI, but the HTTP baseline was faster. Correctness checks matched the PyTorch reference within small tolerances before benchmarking. The authors argue faster scoring mainly frees shared compute for rollout generation, rather than automatically shortening the whole RLHF step. HF Daily Papers' note
score 4