OpenAI finds roughly 30 percent of popular AI coding test is broken
OpenAI reviewed the SWE-Bench Pro benchmark and found about 30% of its tasks are broken, prompting the company to withdraw its endorsement.
- OpenAI identified that about 30% of SWE-Bench Pro tasks are broken.
- The company has withdrawn its endorsement of the benchmark.
- Reliability of AI coding evaluations may need tighter quality controls.
- Developers and researchers should reassess reliance on SWE-Bench Pro for model assessment.
OpenAI conducted an internal review of SWE-Bench Pro, a benchmark that many researchers and developers use to measure the programming abilities of AI models.
The review uncovered that roughly 30 percent of the benchmark's tasks contain errors or are otherwise broken, calling into question the reliability of the results it produces.
Because of these findings, OpenAI is removing its previous endorsement of SWE-Bench Pro and advising the community to treat the benchmark with caution.
The episode underscores the difficulty of keeping evaluation datasets accurate as AI systems become more capable and the field matures.
A flawed benchmark can mislead developers about model performance.
Accurate evaluation tools are essential for product decisions involving AI coding assistants.
Benchmark credibility affects confidence in AI startups that claim coding prowess.
Learning resources that use the benchmark may need to update their curricula.
- SWE-Bench Pro
- A benchmark suite for testing AI models' software engineering and coding abilities.
SU announces launch of artificial intelligence portfolio - Eagle News Online
Natural traces in forensic investigations: Integrating biological trace analysis and artificial intelligence in forensic investigations - Open Access Government
Gov. Pritzker signs nation-leading artificial intelligence safety law - Illinois Business Journal
UN digital tech agency launches initiative to improve trust in AI agents - Reuters
AI ResearchLoop Engineering for Hierarchical Retrieval: Reading a Long Document by Its Table of Contents
Meta says its new AI model is ready to compete on coding
After reentering the AI race with its first in-house Muse Spark model in April, Meta is now opening up the doors to developers with a new model that can plug into AI coding software with the new Meta Model API. Meta says that Muse Spark 1.1 is a "step-change" from the first generation, with improvements based on feedback from developers. The company says it's capable of more advanced coding, including detection and fixing of complex bugs; better supports end-to-end agentic workflows across a range of apps, including multi-agent systems; and has native multimodal perception across images, video
AI ToolsCharacter.AI wants a piece of the microdrama pie
Character.AI launches c.ai Series, a new platform for AI-generated animated microdramas designed for vertical video. The move targets a rapidly growing $26 billion market.
BusinessThe newest entrant in the military’s launch competition isn't actually a launch company
An AI-focused startup secured a Pentagon launch competition slot despite not operating launch vehicles, raising questions about the competition's criteria.
Over-the-top AI campaign ads will test Oregon disclosure law - Oregon Public Broadcasting - OPB
Oregon’s new law requiring disclosure of AI-generated campaign ads will face its first real test as over-the-top political ads flood the state.
AI ToolsGeForce NOW Turns Up the Heat With New GeForce RTX 5080-Powered Toronto Server
Nvidia's GeForce NOW cloud gaming service expands to Toronto with a new server powered by GeForce RTX 5080 graphics cards.
AI ToolsCharacter.ai enters the microdrama arena with its own productions, but with a twist
Character AI is expanding beyond chatbots by producing short-form interactive dramas where users can converse with characters and shape storylines in real time.