SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research
Even the strongest models sometimes rejected simulator recommendations that remained valid despite modeling errors.
Siris tested thirteen frontier LLMs on design problems for LLM serving systems. It paired workloads with the same candidate designs, simulator limitation and recommendation, but a reference simulator supported that recommendation in only one case. Recognizing the limitation did not reliably lead models to the correct design choice. A review turn helped the strongest models recover valid choices but introduced new errors in several others.
HF Daily Papers' note
Siris tested thirteen frontier LLMs on design problems for LLM serving systems. It paired workloads with the same candidate designs, simulator limitation and recommendation, but a reference simulator supported that recommendation in only one case. Recognizing the limitation did not reliably lead models to the correct design choice. A review turn helped the strongest models recover valid choices but introduced new errors in several others.
HF Daily Papers' note
score 5