As conventional AI benchmarking<\/a> techniques prove inadequate, AI builders are turning to more creative ways to assess the capabilities of generative AI models. For one group of developers, that\u2019s Minecraft, the Microsoft-owned sandbox-building game.<\/p>\n The website Minecraft Benchmark<\/a> (or MC-Bench) was developed collaboratively to pit AI models against each other in head-to-head challenges to respond to prompts with Minecraft creations. Users can vote on which model did a better job, and only after voting can they see which AI made each Minecraft build.<\/p>\n For Adi Singh, the 12th grader who started MC-Bench, the value of Minecraft isn\u2019t so much the game itself, but the familiarity that people have with it \u2014 after all, it is the best-selling<\/a> video game of all time. Even for people who haven\u2019t played the game, it\u2019s still possible to evaluate which blocky representation of a pineapple is better realized.<\/p>\n \u201cMinecraft allows people to see the progress [of AI development] much more easily,\u201d Singh told TechCrunch. \u201cPeople are used to Minecraft, used to the look and the vibe.\u201d<\/p>\n MC-Bench currently lists eight people as volunteer contributors. Anthropic, Google, OpenAI, and Alibaba have subsidized the project\u2019s use of their products to run benchmark prompts, per MC-Bench\u2019s website, but the companies are not otherwise affiliated. <\/p>\n \u201cCurrently we are just doing simple builds to reflect on how far we\u2019ve come from the GPT-3 era, but [we] could see ourselves scaling to these longer-form plans and goal-oriented tasks,\u201d Singh said. \u201cGames might just be a medium to test agentic reasoning that is safer than in real life and more controllable for testing purposes, making it more ideal in my eyes.\u201d<\/p>\n Other games like Pok\u00e9mon Red<\/a>,\u00a0Street Fighter<\/a>, and Pictionary<\/a> have been used as experimental benchmarks for AI, in part because the art of benchmarking AI is notoriously tricky<\/a>.<\/p>\n Researchers often test AI models on standardized evaluations<\/a>, but many of these tests give AI a home-field advantage. Because of the way they\u2019re trained, models are naturally gifted at certain, narrow kinds of problem-solving, particularly problem-solving that requires rote memorization or basic extrapolation.<\/p>\n Put simply, it\u2019s hard to glean what it means that OpenAI\u2019s GPT-4 can score in the 88th percentile on the LSAT, but cannot discern how many Rs are in the word \u201cstrawberry.\u201d<\/a> Anthropic\u2019s Claude 3.7 Sonnet<\/a> achieved 62.3% accuracy on a standardized software engineering benchmark, but it is worse at playing Pok\u00e9mon than most five-year-olds. <\/p>\n MC-Bench is technically a programming benchmark, since the models are asked to write code to create the prompted build, like \u201cFrosty the Snowman\u201d or \u201ca charming tropical beach hut on a pristine sandy shore.\u201d<\/p>\n But it\u2019s easier for most MC-Bench users to evaluate whether a snowman looks better than to dig into code, which gives the project wider appeal \u2014 and thus the potential to collect more data about which models consistently score better.<\/p>\n Whether those scores amount to much in the way of AI usefulness is up for debate, of course. Singh asserts that they\u2019re a strong signal, though.<\/p>\n \u201cThe current leaderboard reflects quite closely to my own experience of using these models, which is unlike a lot of pure text benchmarks,\u201d Singh said. \u201cMaybe [MC-Bench] could be useful to companies to know if they\u2019re heading in the right direction.\u201d<\/p>\n<\/div>\n![\"\"](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2025\/03\/Screenshot-2025-03-20-at-2.22.35PM.png?w=680\")
<\/figure>\n