\n<\/p>\n
A Meta exec on Monday denied a rumor that the company trained its new AI models to present well on specific benchmarks while concealing the models\u2019 weaknesses.<\/p>\n
The executive, Ahmad Al-Dahle, VP of generative AI at Meta, said in a post on X<\/a> that it\u2019s \u201csimply not true\u201d that Meta trained its Llama 4 Maverick and Llama 4 Scout models<\/a> on \u201ctest sets.\u201d In AI benchmarks, test sets are collections of data used to evaluate the performance of a model after it\u2019s been trained. Training on a test set could misleadingly inflate a model\u2019s benchmark scores, making the model appear more capable than it actually is.<\/p>\n Over the weekend, an unsubstantiated rumor<\/a> that Meta artificially boosted its new models\u2019 benchmark results began circulating on X and Reddit. The rumor appears to have originated from a post on a Chinese social media site from a user claiming to have resigned from Meta in protest over the company\u2019s benchmarking practices.<\/p>\n Reports that Maverick and Scout perform<\/a> poorly<\/a> on certain tasks<\/a> fueled the rumor, as did Meta\u2019s decision to use an experimental, unreleased version of Maverick<\/a> to achieve better scores on the benchmark LM Arena<\/a>. Researchers on X have\u00a0observed stark<\/a>\u00a0differences in the behavior<\/a>\u00a0of the publicly downloadable Maverick compared with the model hosted on LM Arena.\u00a0<\/p>\n Al-Dahle acknowledged that some users are seeing \u201cmixed quality\u201d from Maverick and Scout across the different cloud providers hosting the models.<\/p>\n \u201cSince we dropped the models as soon as they were ready, we expect it\u2019ll take several days for all the public implementations to get dialed in,\u201d Al-Dahle said. \u201cWe\u2019ll keep working through our bug fixes and onboarding partners.\u201d<\/p>\n<\/div>\n