QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code
QTestBench tests whether LLM-written quantum test suites catch bad code, not just whether they run.
The framework compares generated tests against independent benchmark checkers instead of treating the tests as truth. It prompts models for syntactic, semantic, and behavioral tests, then injects candidate quantum solutions at runtime. Across five LLMs, eight PennyLane QHack challenges, and ten Qiskit HumanEval tasks, execution was high but discrimination was weaker: the best score was 0.71 on QHack and 0.34 on Qiskit HumanEval. The paper says synthetic mutants help estimate bug detection when natural faulty solutions are scarce. ArXiv · AI/CL/LG's note
The framework compares generated tests against independent benchmark checkers instead of treating the tests as truth. It prompts models for syntactic, semantic, and behavioral tests, then injects candidate quantum solutions at runtime. Across five LLMs, eight PennyLane QHack challenges, and ten Qiskit HumanEval tasks, execution was high but discrimination was weaker: the best score was 0.71 on QHack and 0.34 on Qiskit HumanEval. The paper says synthetic mutants help estimate bug detection when natural faulty solutions are scarce. ArXiv · AI/CL/LG's note
score 4