OpenAI believes that AI benchmarks are flawed and is now launching a program to improve how AI models are evaluated. The new OpenAI Pioneers Program aims to establish standards for what defines high-quality AI models, as stated in a blog post.

“With the rapid increase in AI adoption across various industries, it’s crucial to understand and enhance its impact on the world,” the company explained. “Developing domain-specific evaluations is a way to better represent real-world scenarios, enabling teams to assess model performance in practical and high-stakes situations.”

Recent issues with benchmarks like LM Arena and Meta’s Maverick model highlight the challenge in distinguishing between different AI models. Many existing benchmarks assess performance on complex tasks, such as solving advanced math problems, which may not align with common preferences or can be manipulated.

Through the Pioneers Program, OpenAI plans to create benchmarks tailored for specific sectors like legal, finance, insurance, healthcare, and accounting. The lab intends to collaborate with multiple companies to design customized benchmarks and share them publicly, along with industry-specific assessments.

“The initial focus of the program will be on startups that can help establish the foundation of the OpenAI Pioneers Program,” OpenAI stated. “We will select a few startups for the first cohort, each working on practical applications where AI can make a real impact.”

Participants in the program will have the chance to collaborate with OpenAI to enhance models through reinforcement fine-tuning, a technique that optimizes models for specific tasks.

The key question is whether the AI community will accept benchmarks funded by OpenAI. While OpenAI has supported benchmarking financially in the past and developed its own evaluations, working with customers to release AI tests may raise ethical concerns.