Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
The paper says local tool-use scores can be artifacts of the serving stack, not the model.
Tang and Zheng show that Ollama handles the same `tools=` request differently by model, including rejecting Phi-3 and Gemma-3 before inference. Their harness can then lose the rejection metadata, letting downstream analysis mistake serving failures for model non-calls. They also find that protocol choices change measured fidelity: adding a text tool list helps some accepted models, while a text-only protocol hurts Llama-3.2. Cross-stack checks on Ollama, vLLM, and SGLang produced different behavior for the same request. ArXiv · AI/CL/LG's note
Tang and Zheng show that Ollama handles the same `tools=` request differently by model, including rejecting Phi-3 and Gemma-3 before inference. Their harness can then lose the rejection metadata, letting downstream analysis mistake serving failures for model non-calls. They also find that protocol choices change measured fidelity: adding a text tool list helps some accepted models, while a text-only protocol hurts Llama-3.2. Cross-stack checks on Ollama, vLLM, and SGLang produced different behavior for the same request. ArXiv · AI/CL/LG's note
score 4