\n<\/p>\n
Last month, I wrote about Mercor\u2019s new benchmark<\/a> measuring AI agents\u2019 capabilities on professional tasks like law and corporate analysis. At the time, the scores were pretty dismal, with every major lab scoring under 25%, so we concluded lawyers were safe from AI displacement, at least for now.<\/p>\n But AI capabilities can change a lot in a couple of weeks.<\/p>\n This week\u2019s release of Anthropic\u2019s Opus 4.6<\/a> shook up the leaderboards<\/a>, with Anthropic\u2019s new model scoring just shy of 30% in one-shot trials, and an average of 45% when given a few more cracks at the problem. Notably, the release included a bunch of new agentic features, including \u201cagent swarms,\u201d which may have helped with this kind of multistep problem-solving.<\/p>\n Regardless, the score is a huge jump from the previous state-of-the-art, and a sign that progress on foundation models isn\u2019t slowing down. Mercor CEO Brendan Foody, who was particularly impressed, said, \u201cjumping from 18.4% to 29.8% in a few months is insane.\u201d<\/p>\n Thirty percent is still a long way from 100%, so it\u2019s not like lawyers need to be worried about getting replaced by machines next week. But they should be a lot less confident than they were last month!<\/p>\n<\/div>\n