\n<\/p>\n
The Wikimedia Foundation, the umbrella organization of Wikipedia and a dozen or so other<\/a> crowdsourced knowledge projects, said on Wednesday that bandwidth consumption for multimedia downloads from Wikimedia Commons<\/a> has surged by 50% since January 2024.<\/p>\n The reason, the outfit wrote in a blog post<\/a> Tuesday, isn\u2019t due to growing demand from knowledge-thirsty humans, but from automated, data-hungry scrapers looking to train AI models. <\/p>\n \u201cOur infrastructure is built to sustain sudden traffic spikes from humans during high-interest events, but the amount of traffic generated by scraper bots is unprecedented and presents growing risks and costs,\u201d the post reads. <\/p>\n Wikimedia Commons is a freely accessible repository of images, videos and audio files that are available under open licenses or are otherwise in the public domain.<\/p>\n Digging down, Wikimedia says that almost two-thirds (65%) of the most \u201cexpensive\u201d traffic \u2014 that is, the most resource-intensive in terms of the kind of content consumed \u2014 was from bots. However, just 35% of the overall pageviews comes from these bots. The reason for this disparity, according to Wikimedia, is that frequently-accessed content stays closer to the user in its cache, while other less-frequently accessed content is stored further away in the \u201ccore data center,\u201d which is more expensive to serve content from. This is the kind of content that bots typically go looking for.<\/p>\n \u201cWhile human readers tend to focus on specific \u2013 often similar \u2013 topics, crawler bots tend to \u2018bulk read\u2019 larger numbers of pages and visit also the less popular pages,\u201d Wikimedia writes. \u201cThis means these types of requests are more likely to get forwarded to the core datacenter, which makes it much more expensive in terms of consumption of our resources.\u201d<\/p>\n The long and short of all this is that the Wikimedia Foundation\u2019 site reliability team are having to spend a lot of time and resources blocking crawlers to avert disruption for regular users. And all this before we consider the cloud costs that the Foundation is faced with. <\/p>\n In truth, this represents part of a fast-growing trend that is threatening the very existence of the open internet. Last month, software engineer and open source advocate \u200bDrew DeVault bemoaned the fact<\/a> that AI crawlers ignore \u201crobots.txt\u201d files that are designed to ward off automated traffic. And \u201cpragmatic engineer<\/a>\u201d Gergely Orosz also complained<\/a> last week that AI scrapers from companies such as Meta have driven up bandwidth demands for his own projects.<\/p>\n While open source infrastructure, in particular, is in the firing line<\/a>, developers are fighting back with \u201ccleverness and vengeance,\u201d as TechCrunch wrote last week<\/a>. Some tech companies are doing their bit to address the issue, too \u2014 Cloudflare, for example, recently launched AI Labyrinth<\/a>, which uses AI-generated content to slow crawlers down.<\/p>\n However, it\u2019s very much a cat-and-mouse game that could ultimately force many publishers to duck for cover behind logins and paywalls \u2014 to the detriment of everyone who uses the web today<\/a>.<\/p>\n<\/div>\n