OpenAI wants new AI benchmarks to replace Scale's "broken" tests

"We retract our earlier recommendation to adopt SWE-Bench Pro."

Updated Jul 9, 2026 4:52 PM / Published Jul 9, 2026 11:29 AM 2 min read

OpenAI wants new AI benchmarks to replace Scale's "broken" tests — Image credit: https://unsplash.com/@joshua_hoehne

OpenAI is calling for developers to ditch SWE-Bench Pro, one of the most popular AI benchmarks, and build a replacement, claiming almost 30% of its tests are “broken."

As it released its new GPT-5.6 Sol model on Thursday, OpenAI said it had also audited the Scale AI-developed benchmark and found a series of issues, despite recommending its use in February 2026.

In a July 8 blog, OpenAI’s research team retracted its recommendation and said: “We hope the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities.”

This content is for members only

Already have an account? Sign in

Add The Stack on Google

Noah Bovenizer

Noah Bovenizer is a senior reporter with The Stack. He previously covered the transportation sector for Global Data. He has a first class degree in journalism and has also worked as a newsreader for Gaydio.