\n<\/p>\n
OpenAI, like many AI labs, thinks AI benchmarks are broken. It says it wants to fix them through a new program.<\/p>\n
Called the OpenAI Pioneers Program, the program will focus on creating evaluations for AI models that \u201cset the bar for what good looks like,\u201d as OpenAI phrased it in a blog post<\/a>. <\/p>\n \u201cAs the pace of AI adoption accelerates across industries, there is a need to understand and improve its impact in the world,\u201d the company continued in its post. \u201cCreating domain-specific evals are one way to better reflect real-world use cases, helping teams assess model performance in practical, high-stakes environments.\u201d<\/p>\n As the recent<\/a> controversy<\/a> with the crowdsourced benchmark LM Arena and Meta\u2019s Maverick model illustrate, it\u2019s tough to know, these days, precisely what differentiates one model from another. Many widely-used AI benchmarks measure performance on esoteric tasks, like solving doctorate-level math problems. Others can be gamed, or don\u2019t align well with most people\u2019s preferences. <\/p>\n Through the Pioneers Program, OpenAI hopes to create benchmarks for specific domains like legal, finance, insurance, healthcare, and accounting. The lab says that, in the coming months, it\u2019ll work with \u201cmultiple companies\u201d to design tailored benchmarks and eventually share those benchmarks publicly, along with \u201cindustry-specific\u201d evaluations. <\/p>\n \u201cThe first cohort will focus on startups who will help lay the foundations of the OpenAI Pioneers Program,\u201d OpenAI wrote in the blog post. \u201cWe\u2019re selecting a handful of startups for this initial cohort, each working on high-value, applied use cases where AI can drive real-world impact.\u201d<\/p>\n Companies in the program will also have the opportunity to work with OpenAI\u2019s team to create model improvements via reinforcement fine tuning, a technique that optimizes models for a narrow set of tasks, OpenAI says. <\/p>\n The big question is whether the AI community will embrace benchmarks whose creation was funded by OpenAI. OpenAI has supported benchmarking efforts financially before, and designed its own evaluations. But partnering with customers to release AI tests may be seen as an ethical bridge too far.<\/p>\n<\/div>\n