Simplicity Goes Far: Auditing Prompt Optimizers with Demonstration-Conditioned Prompt Search
A simple generate-and-select baseline matched two heavier prompt optimizers at substantially lower measured cost.
Demonstration-Conditioned Prompt Search freezes the generator, uses randomly sampled demonstrations to propose prompts, and keeps the best validation-scoring candidate. Replacing PPO in StablePrompt, it matched classification accuracy within 0.5 percentage points at roughly one-third the measured cost. In DSPy systems using GPT-4.1-mini, it matched GEPA’s macro accuracy under the reported budgets, with 55% fewer rollouts and 32% lower measured API cost on average. Additional optimizer mechanisms retained advantages in some open-ended and reasoning-heavy settings. The paper recommends generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy.
ArXiv · AI/CL/LG's note
Demonstration-Conditioned Prompt Search freezes the generator, uses randomly sampled demonstrations to propose prompts, and keeps the best validation-scoring candidate. Replacing PPO in StablePrompt, it matched classification accuracy within 0.5 percentage points at roughly one-third the measured cost. In DSPy systems using GPT-4.1-mini, it matched GEPA’s macro accuracy under the reported budgets, with 55% fewer rollouts and 32% lower measured API cost on average. Additional optimizer mechanisms retained advantages in some open-ended and reasoning-heavy settings. The paper recommends generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy.
ArXiv · AI/CL/LG's note
score 5