\n<\/p>\n
An organization OpenAI frequently partners with to probe the capabilities of its AI models and evaluate them for safety, Metr, suggests that it wasn\u2019t given much time to test one of the company\u2019s highly capable new releases, o3<\/a>. <\/p>\n In a blog post published Wednesday<\/a>, Metr writes that one red teaming benchmark of o3 was \u201cconducted in a relatively short time\u201d compared to the organization\u2019s testing of a previous OpenAI flagship model, o1<\/a>. This is significant, they say, because additional testing time can lead to more comprehensive results.<\/p>\n \u201cThis evaluation was conducted in a relatively short time, and we only tested [o3] with simple agent scaffolds,\u201d wrote Metr in its blog post. \u201cWe expect higher performance [on benchmarks] is possible with more elicitation effort.\u201d<\/p>\n Recent reports suggest that OpenAI, spurred by competitive pressure, is rushing independent evaluations. According to the Financial Times<\/a>, OpenAI gave some testers less than a week for safety checks for an upcoming major launch.<\/p>\n In statements, OpenAI has disputed the notion that it\u2019s compromising on safety.<\/p>\n Metr says that, based on the information it was able to glean in the time it had, o3 has a \u201chigh propensity\u201d to \u201ccheat\u201d or \u201chack\u201d tests in sophisticated ways in order to maximize its score \u2014 even when the model clearly understands its behavior is misaligned with the user\u2019s (and OpenAI\u2019s) intentions. The organization thinks it\u2019s possible o3 will engage in other types of adversarial or \u201cmalign\u201d behavior, as well \u2014 regardless of the model\u2019s claims to be aligned, \u201csafe by design,\u201d or not have any intentions of its own.<\/p>\n \u201cWhile we don\u2019t think this is especially likely, it seems important to note that [our] evaluation setup would not catch this type of risk,\u201d Metr wrote in its post. \u201cIn general, we believe that pre-deployment capability testing is\u00a0not a sufficient risk management strategy\u00a0by itself, and we are currently prototyping additional forms of evaluations.\u201d<\/p>\n Another of OpenAI\u2019s third-party evaluation partners, Apollo Research, also observed deceptive behavior from o3 and the company\u2019s other new model, o4-mini. In one test, the models, given 100 computing credits for an AI training run and told not to modify the quota, increased the limit to 500 credits \u2014 and lied about it. In another test, asked to promise not to use a specific tool, the models used the tool anyway when it proved helpful in completing a task. <\/p>\n