Separating signal from noise in coding evaluations
OpenAI identified reliability problems in SWE-Bench Pro, weakening confidence in a prominent coding benchmark for AI model evaluation.
Excerpt
A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
Read at source: https://openai.com/index/separating-signal-from-noise-coding-evaluations