\n<\/p>\n
On Tuesday, OpenAI revealed that one of its models went rogue during a test and hacked the systems of AI dataset platform Hugging Face in a fully AI-enabled attack<\/a>, a dramatic example of the dangers posed by advanced AI models.<\/p>\n But, according to some cybersecurity experts, at the heart of this unprecedented AI-powered breach there was a very human mistake: OpenAI failed to properly configure what it called<\/a> a \u201chighly isolated environment,\u201d allowing a testing sandbox that should have been completely secluded from the internet to actually connect to the internet.\u00a0<\/p>\n Dan Guido, the founder of cybersecurity research startup Trail of Bits, called the mistake \u201ca containment failure with the safeties turned off.\u201d<\/p>\n In its blog post detailing the incident<\/a>, OpenAI said that the test that led to the Hugging Face breach was set up to run in \u201ca highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.\u201d\u00a0<\/p>\n The model was able to escape the sandboxed testing environment thanks to a previously undisclosed vulnerability in the package-installation system, a critical first step in the eventual hack on Hugging Face, according to OpenAI.<\/p>\n In response, the company \u201cresponsibly disclosed the identified zero-day vulnerability in the internally-hosted third-party software and are working with them to patch.\u201d\u00a0<\/p>\n But to most cybersecurity professionals, software vulnerabilities are to be expected \u2014 and the real fault lies with the decision to maintain the third-party software in the first place. Ultimately, the value of a \u201csandbox\u201d system lies in its full and total isolation. Including a package-installation system is asking for trouble.<\/p>\n Martin Boone, a cybersecurity researcher, told TechCrunch that \u201cthis sounds like human failure.\u201d<\/p>\n \u201cThis should never have happened,\u201d Boone said. \u201cIf sandbox would actually mean sandbox, you expect it to have no physical connection to the internet whatsoever. This sounds more like they had some firewalling or something in place, and firewalling is hard from the outside in, let alone inside to the outside internet.\u201d<\/p>\n Cybersecurity veteran Jake Williams agreed. \u201cAny model performing the types of actions documented by Hugging Face was not fully contained in a sandbox,\u201d said Williams, who called this \u201ca massive control failure\u201d by OpenAI.\u00a0<\/p>\n \u201cOne man\u2019s \u2018the model escaped the sandbox\u2019 is another man\u2019s \u2018you failed to build the sandbox correctly, so of course it escaped,\u2019\u201d Williams continued.<\/p>\n \t\t\tDo you have more information about this incident? Or about other AI-enabled cyberattacks? We\u2019d love to hear from you. From a non-work device and network, you can contact Lorenzo Franceschi-Bicchierai securely on Signal at +1 917 257 1382, or via Telegram and Keybase @lorenzofb, or email<\/a>.<\/a>\t\t<\/div>\n Daniel Card, a cybersecurity consultant, agreed that OpenAI \u201cdidn\u2019t put adequate effort into the design of the sandbox nor its controls\u201d by giving the sandbox or some part of it \u201can unfiltered route to the internet.\u201d Setting up the sandbox, even with limited network access as OpenAI described it, was not a \u201creasonable\u201d decision, according to Card.\u00a0<\/p>\n To be sure, those criticisms have the benefit of hindsight, but they raise real questions about security practices in AI labs \u2014 particularly in maintaining isolated environments for testing models. OpenAI spokespeople did not respond to TechCrunch\u2019s questions, which included whether an AI or a human had set up the testing environment.<\/p>\n But those questions go far beyond OpenAI.\u00a0<\/p>\n In the document<\/a> introducing its cybersecurity-focused model Mythos<\/a>, Anthropic wrote that in a test, the model \u201cwas provided with a secured \u2018sandbox\u2019 computer to interact with,\u201d and instructed to try to escape that \u201csecure container.\u201d Mythos succeeded and gained broader access to the internet \u201cfrom a system that was meant to be able to reach only a small number of predetermined services.\u201d Still, Anthropic noted that the model was not able to \u201cfully\u201d escape the designed containment.<\/p>\n<\/div>\n When you purchase through links in our articles, we may earn a small commission<\/a>. This doesn\u2019t affect our editorial independence.<\/em><\/p>\nContact Us<\/h4>\n