Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
Temperature-zero semantic queries still changed answers across repeat runs.
The paper reruns SemBench, LRO-Bench, and UDA-Bench workloads and logs 1.2 million model calls. Depending on the model, 0.07% to 10.2% of rows changed operator output over five executions, with one task reaching 24.0%. Small row-level changes compounded: on UDA-Bench, a few percent of extracted values changed, but about two thirds of final answers did. Voting reduced variance but did not eliminate it. Source: ArXiv · AI/CL/LG's note.
The paper reruns SemBench, LRO-Bench, and UDA-Bench workloads and logs 1.2 million model calls. Depending on the model, 0.07% to 10.2% of rows changed operator output over five executions, with one task reaching 24.0%. Small row-level changes compounded: on UDA-Bench, a few percent of extracted values changed, but about two thirds of final answers did. Voting reduced variance but did not eliminate it. Source: ArXiv · AI/CL/LG's note.
score 5