RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics
RATED re-ranks agent tools using live execution feedback, not just semantic fit.
The paper proposes a retrieval system that pairs a frozen, contrastively tuned dual encoder with a lightweight contextual bandit. Its aim is to steer agents away from tools that are failing at runtime without retraining the retriever. The authors introduce EcoAgent, with 5,172 real-world tools and more than 210,000 query-tool pairs, to test both offline retrieval and reliability under failures. They report a 52.2% relative MRR gain over the ToolLLM API retriever and a drop in fault selection from 86.2% to 2.9% during severe API outages, with less than 1 ms added latency. ArXiv · AI/CL/LG's note
The paper proposes a retrieval system that pairs a frozen, contrastively tuned dual encoder with a lightweight contextual bandit. Its aim is to steer agents away from tools that are failing at runtime without retraining the retriever. The authors introduce EcoAgent, with 5,172 real-world tools and more than 210,000 query-tool pairs, to test both offline retrieval and reliability under failures. They report a 52.2% relative MRR gain over the ToolLLM API retriever and a drop in fault selection from 86.2% to 2.9% during severe API outages, with less than 1 ms added latency. ArXiv · AI/CL/LG's note
score 5