A major copyright lawsuit against Meta has revealed a trove of internal communications about the company\u2019s plans to develop its open-source AI models, Llama, which include discussions about avoiding \u201cmedia coverage suggesting we have used a dataset we know to be pirated.\u201d<\/p>\n<\/div>\n
The messages, which were part of a series of exhibits unsealed by a California court, suggest Meta used copyrighted data when training its AI systems and worked to conceal it \u2014 as it raced to beat rivals like OpenAI and Mistral. Portions of the messages were first revealed<\/a> last week.<\/p>\n<\/div>\n In an October 2023 email to Meta AI researcher Hugo Touvron, Ahmad Al-Dahle, Meta\u2019s vice president of generative AI, wrote that the company\u2019s goal<\/a> \u201cneeds to be GPT4,\u201d referring to the large language model OpenAI announced in March of 2023<\/a>. Meta had \u201cto learn how to build frontier and win this race,\u201d Al-Dahle added. Those plans apparently involved the book piracy site Library Genesis (LibGen)<\/a> to train its AI systems. <\/p>\n<\/div>\n An undated email from Meta director of product Sony Theakanath<\/a>, sent to VP of AI research Joelle Pineau, weighed whether to use LibGen internally only, for benchmarks included in a blog post, or to create a model trained on the site. In the email, Theakanath writes that \u201cGenAI has been approved to use LibGen for Llama3… with a number of agreed upon mitigations\u201d after escalating it to \u201cMZ\u201d \u2014 presumably Meta CEO Mark Zuckerberg. As noted in the email, Theakanath believed \u201cLibgen is essential to meet SOTA [state-of-the-art] numbers,\u201d adding \u201cit is known that OpenAI and Mistral are using the library for their models (through word of mouth).\u201d Mistral and OpenAI haven\u2019t stated whether or not they use LibGen. (The Verge<\/em> reached out to both for more information).<\/p>\n<\/div>\n The court documents stem from a class action lawsuit<\/a> that author Richard Kadrey, comedian Sarah Silverman, and others filed against Meta, accusing it of using illegally obtained copyrighted content to train its AI models in violation of intellectual property laws. Meta, like other AI companies, has argued that using copyrighted material in training data should constitute legal fair use. The Verge<\/em> reached out to Meta with a request for comment but didn\u2019t immediately hear back.<\/p>\n<\/div>\n Some of the \u201cmitigations\u201d for using LibGen included stipulations that Meta must \u201cremove data clearly marked as pirated\/stolen,\u201d while avoiding externally citing \u201cthe use of any training data\u201d from the site. Theakanath\u2019s email also said the company would need to \u201cred team\u201d the company\u2019s models \u201cfor bioweapons and CBRNE [Chemical, Biological, Radiological, Nuclear, and Explosives]\u201d risks.<\/p>\n<\/div>\n The email also went over some of the \u201cpolicy risks\u201d posed by the use of LibGen as well, including how regulators might respond to media coverage suggesting Meta\u2019s use of pirated content. \u201cThis may undermine our negotiating position with regulators on these issues,\u201d the email said. An April 2023 conversation<\/a> between Meta researcher Nikolay Bashlykov and AI team member David Esiobu also showed Bashlykov admitting he\u2019s \u201cnot sure we can use meta\u2019s IPs to load through torrents [of] pirate content.\u201d<\/p>\n<\/div>\n Other internal documents<\/a> show the measures Meta took to obscure the copyright information in LibGen\u2019s training data. A document titled \u201cobservations on LibGen-SciMag\u201d shows comments left by employees about how to improve the dataset. One suggestion is to \u201cremove more copyright headers and document identifiers,\u201d which includes any lines containing \u201cISBN,\u201d \u201cCopyright,\u201d \u201cAll rights reserved,\u201d or the copyright symbol. Other notes mention taking out more metadata \u201cto avoid potential legal complications,\u201d as well as considering whether to remove a paper\u2019s list of authors \u201cto reduce liability.\u201d<\/p>\n<\/div>\n Last June, The New York Times <\/em>reported<\/a> on the frantic race inside Meta after ChatGPT\u2019s debut, revealing the company had hit a wall: it had used up almost every available English book, article, and poem it could find online. Desperate for more data, executives reportedly discussed buying Simon & Schuster outright and considered hiring contractors in Africa to summarize books without permission. <\/p>\n<\/div>\n In the report, some executives justified their approach by pointing to OpenAI\u2019s \u201cmarket precedent\u201d of using copyrighted works, while others argued Google\u2019s 2015 court victory establishing its right to scan books<\/a> could provide legal cover. \u201cThe only thing holding us back from being as good as ChatGPT is literally just data volume,\u201d one executive said in a meeting, per The New York Times<\/em>.<\/p>\n<\/div>\n It\u2019s been reported that frontier labs like OpenAI and Anthropic have hit a data wall, which means they don\u2019t have sufficient new data to train their large language models. Many leaders have denied this, OpenAI CEO Sam Altman said plainly<\/a>: \u201cThere is no wall.\u201d OpenAI cofounder Ilya Sutskever, who left the company last May<\/a> to start a new frontier lab, has been more straightforward about the potential of a data wall. At a premier AI conference last month<\/a>, Sutskever said: \u201cWe\u2019ve achieved peak data and there\u2019ll be no more. We have to deal with the data that we have. There\u2019s only one internet.\u201d<\/p>\n<\/div>\n This data scarcity has led to a whole lot of weird, new ways to get unique data. Bloomberg <\/em>reported<\/a> that frontier labs like OpenAI and Google have been paying digital content creators between $1 and $4 per minute for their unused video footage through a third-party in order to train LLMs (both of those companies have competing AI video-generation products).<\/p>\n<\/div>\n With companies like Meta and OpenAI hoping to grow their AI systems as fast as possible, things are bound to get a bit messy. Though a judge partially dismissed Kadrey and Silverman\u2019s class action<\/a> lawsuit last year, the evidence outlined here could strengthen parts of their case as it moves forward in court.<\/p>\n<\/div>\n<\/div>\n