OpenAI wants new AI benchmarks to replace Scale's "broken" tests

OpenAI is calling for developers to ditch SWE-Bench Pro, one of the most popular AI benchmarks, and build a replacement, claiming almost 30% of its tests are “broken”.

As it released its new GPT-5.6 Sol model on Thursday, OpenAI said it had also audited the Scale AI-developed benchmark and found a series of issues, despite recommending its use in February 2026.

In a July 8 blog, OpenAI’s research team retracted its recommendation and said: “We hope the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities.”