\n<\/p>\n
OpenAI admitted Tuesday that one of its AI models breached Hugging Face\u2019s systems during an internal cybersecurity test that went awry. Hugging Face initially attributed the breach<\/a> to an \u201cexternal AI agent.\u201d<\/p>\n In a blog post published Tuesday afternoon<\/a>, OpenAI detailed the steps that led the models to compromise the service.<\/p>\n \u201cAfter investigating, we now know that this particular incident was driven by a combination of OpenAI models \u2014 including GPT\u20115.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes \u2014 while being internally tested on a benchmark\u2060 of cyber capabilities,\u201d the post reads.<\/p>\n In particular, the breach appears to have focused on ExploitGym<\/a>, a publicly hosted benchmark measuring models\u2019 ability to execute attacks based on existing vulnerabilities. Benchmarks like ExploitGym are commonly used in model training to refine specific skills, but this is the first known incident in which that testing resulted in an actual cyberattack.<\/p>\n In this case, the model in question should not have even had internet access, outside of a specific tool that enabled models to install software packages they might need to complete their task. Instead, the model was able to find an undisclosed vulnerability in the package-installer program, which it used to access the broader internet at will.<\/p>\n \u201cThe models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,\u201d OpenAI\u2019s post reads. \u201cAfter gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.\u201d<\/p>\n Ultimately, the models found vulnerabilities in Hugging Face\u2019s infrastructure that allowed them to \u201cobtain test solutions directly from Hugging Face\u2019s production database,\u201d effectively providing the answers to the benchmark.<\/p>\n For Hugging Face, the apparent result was a sophisticated and aggressive cyberattack, with \u201cmany thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services,\u201d as the company stated in its initial disclosure.<\/p>\n OpenAI has identified and reported the vulnerabilities in the package installer and is working with Hugging Face to investigate the incident further. The company also said it would implement new controls on both model testing and the related infrastructure, meant to prevent similar incidents in the future.<\/p>\n It\u2019s unclear whether OpenAI will face any legal consequences as a result of the breach, although it\u2019s likely that the models\u2019 actions violated the Computer Fraude and Abuse Act.<\/p>\n Nevertheless, the result is an unusually vivid illustration of the power and dangers of frontier AI models operating on long time horizons. As OpenAI researcher Micah Carroll posted in response to the news<\/a>, \u201cIf this doesn\u2019t convince you that misalignment risks are going to be a key concern going forward, I don\u2019t know what will.\u201d<\/p>\n<\/div>\n When you purchase through links in our articles, we may earn a small commission<\/a>. This doesn\u2019t affect our editorial independence.<\/em><\/p>\n