\n<\/p>\n
A new paper<\/a> from AI lab Cohere, Stanford, MIT, and Ai2 accuses LM Arena, the organization behind the popular crowdsourced AI benchmark Chatbot Arena, of helping a select group of AI companies achieve better leaderboard scores at the expense of rivals.<\/p>\n According to the authors, LM Arena allowed some industry-leading AI companies like Meta, OpenAI, Google, and Amazon to privately test several variants of AI models, then not publish the scores of the lowest performers. This made it easier for these companies to achieve a top spot on the platform\u2019s leaderboard, though the opportunity was not afforded to every firm, the authors say.<\/p>\n \u201cOnly a handful of [companies] were told that this private testing was available, and the amount of private testing that some [companies] received is just so much more than others,\u201d said Cohere\u2019s VP of AI research and co-author of the study, Sara Hooker, in an interview with TechCrunch. \u201cThis is gamification.\u201d<\/p>\n Created in 2023 as an academic research project out of UC Berkeley, Chatbot Arena has become a go-to benchmark for AI companies. It works by putting answers from two different AI models side-by-side in a \u201cbattle,\u201d and asking users to choose the best one. It\u2019s not uncommon to see unreleased models competing in the arena under a pseudonym. <\/p>\n Votes over time contribute to a model\u2019s score \u2014 and, consequently, its placement on the Chatbot Arena leaderboard. While many commercial actors participate in Chatbot Arena, LM Arena has long maintained that its benchmark is an impartial and fair one.<\/p>\n However, that\u2019s not what the paper\u2019s authors say they uncovered.<\/p>\n One AI company, Meta, was able to privately test 27 model variants on Chatbot Arena between January and March leading up to the tech giant\u2019s Llama 4 release, the authors allege. At launch, Meta only publicly revealed the score of a single model \u2014 a model that happened to rank near the top of the Chatbot Arena leaderboard.<\/p>\n Techcrunch event<\/p>\n \n\t\t\t\t\t\t\t\t\tBerkeley, CA<\/span> \t\t\t\t\t\t\t In an email to TechCrunch, LM Arena Co-Founder and UC Berkeley Professor Ion Stoica said that the study was full of \u201cinaccuracies\u201d and \u201cquestionable analysis.\u201d<\/p>\n \u201cWe are committed to fair, community-driven evaluations, and invite all model providers to submit more models for testing and to improve their performance on human preference,\u201d said LM Arena in a statement provided to TechCrunch.\u00a0\u201cIf a model provider chooses to submit more tests than another model provider, this does not mean the second model provider is treated unfairly.\u201d<\/p>\n The paper\u2019s authors started conducting their research in November 2024 after learning that some AI companies were possibly being given preferential access to Chatbot Arena. In total, they measured more than 2.8 million Chatbot Arena battles over a five-month stretch.<\/p>\n The authors say they found evidence that LM Arena allowed certain AI companies, including Meta, OpenAI, and Google, to collect more data from Chatbot Arena by having their models appear in a higher number of model \u201cbattles.\u201d This increased sampling rate gave these companies an unfair advantage, the authors allege.<\/p>\n Using additional data from LM Arena could improve a model\u2019s performance on Arena Hard, another benchmark LM Arena maintains, by 112%. However, LM Arena said in a\u00a0post on X<\/a>\u00a0that Arena Hard performance does not directly correlate to Chatbot Arena performance.<\/p>\n Hooker said it\u2019s unclear how certain AI companies might\u2019ve received priority access, but that it\u2019s incumbent on LM Arena to increase its transparency regardless.<\/p>\n In\u00a0a post on X<\/a>, LM Arena said that several of the claims in the paper don\u2019t reflect reality. The organization pointed to a\u00a0blog post<\/a> it published earlier this week indicating that models from non-major labs appear in more Chatbot Arena battles than the study suggests. <\/p>\n One important limitation of the study is that it relied on \u201cself-identification\u201d to determine which AI models were in private testing on Chatbot Arena. The authors prompted AI models several times about their company of origin, and relied on the models\u2019 answers to classify them \u2014 a method that isn\u2019t foolproof. <\/p>\n However, Hooker said that when the authors reached out to LM Arena to share their preliminary findings, the organization didn\u2019t dispute them.<\/p>\n TechCrunch reached out to Meta, Google, OpenAI, and Amazon \u2014 all of which were mentioned in the study \u2014 for comment. None immediately responded.<\/p>\n In the paper, the authors call on LM Arena to implement a number of changes aimed at making Chatbot Arena more \u201cfair.\u201d For example, the authors say, LM Arena could set a clear and transparent limit on the number of private tests AI labs can conduct, and publicly disclose scores from these tests.<\/p>\n In a\u00a0post on X,<\/a>\u00a0LM Arena rejected these suggestions, claiming it has published information on pre-release testing\u00a0since March 2024<\/a>. The benchmarking organization also said it \u201cmakes no sense to show scores for pre-release models which are not publicly available,\u201d because the AI community cannot test the models for themselves.<\/p>\n The researchers also say LM Arena could adjust Chatbot Arena\u2019s sampling rate to ensure that all models in the arena appear in the same number of battles. LM Arena has been receptive to this recommendation publicly, and indicated that it\u2019ll create a new sampling algorithm.<\/p>\n The paper comes weeks after Meta was caught gaming benchmarks in Chatbot Arena around the launch of its above-mentioned Llama 4 models. Meta optimized one of the Llama 4 models for \u201cconversationality,\u201d which helped it achieve an impressive score on Chatbot Arena\u2019s leaderboard. But the company never released the optimized model \u2014 and the vanilla version\u00a0ended up performing much worse<\/a> on Chatbot Arena.<\/p>\n At the time, LM Arena said Meta should have been more transparent in its approach to benchmarking.<\/p>\n
\n\t\t\t\t\t\t\t\t\t\t\t\t\t|<\/span>
\n\t\t\t\t\t\t\t\t\t\t\t\t\tJune 5<\/span>\n\t\t\t\t\t\t\t<\/p>\n
\n\t\t\t\t\tBOOK NOW<\/span>
\n\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\nSupposedly favored labs<\/h2>\n
LM Arena in hot water<\/h2>\n