\n<\/p>\n
A feature Google demoed at its I\/O confab yesterday<\/a>, using its generative AI technology to scan voice calls in real-time for conversational patterns associated with financial scams, has sent a collective shiver down the spines of privacy and security experts who are warning the feature represents the thin end of the wedge. They warn that, once client-side scanning is baked into mobile infrastructure, it could usher in an era of centralized censorship.<\/p>\n Google\u2019s demo of the call scam-detection feature, which the tech giant said would be built into a future version of its Android OS \u2014 estimated to run on some three-quarters of the world\u2019s smartphones \u2014 is powered by Gemini Nano<\/a>, the smallest of its current generation of AI models meant to run entirely on-device.<\/p>\n This is essentially client-side scanning: A nascent technology that\u2019s generated huge controversy in recent years in relation to efforts to detect child sexual abuse material (CSAM) or even grooming activity on messaging platforms. <\/p>\n Apple abandoned a plan to deploy client-side scanning for CSAM in 2021<\/a> after a huge privacy backlash. However policymakers have continued to heap pressure<\/a> on the tech industry to find ways to detect illegal activity taking place on their platforms. Any industry moves to build out on-device scanning infrastructure could therefore pave the way for all-sorts of content scanning by default \u2014 whether government-led, or related to a particular commercial agenda.<\/p>\n Responding to Google\u2019s call scanning demo in a post on X<\/a>, Meredith Whittaker, president of the US-based encrypted messaging app Signal, warned: \u201cThis is incredibly dangerous. It lays the path for centralized, device-level client side scanning.<\/p>\n \u201cFrom detecting \u2018scams\u2019 it\u2019s a short step to \u2018detecting patterns commonly associated w[ith] seeking reproductive care\u2019 or \u2018commonly associated w[ith] providing LGBTQ resources\u2019 or \u2018commonly associated with tech worker whistleblowing\u2019.\u201d<\/p>\n Cryptography expert Matthew Green, a professor at Johns Hopkins, also took to X<\/a> to raise the alarm. \u201cIn the future, AI models will run inference on your texts and voice calls to detect and report illicit behavior,\u201d he warned. \u201cTo get your data to pass through service providers, you\u2019ll need to attach a zero-knowledge proof that scanning was conducted. This will block open clients.\u201d<\/p>\n Green suggested this dystopian future of censorship by default is only a few years out from being technically possible. \u201cWe\u2019re a little ways from this tech being quite efficient enough to realize, but only a few years. A decade at most,\u201d he suggested.<\/p>\n European privacy and security experts were also quick to object.<\/p>\n Reacting to Google\u2019s demo on X<\/a>, Lukasz Olejnik, a Poland-based independent researcher and consultant for privacy and security issues, welcomed the company\u2019s anti-scam feature but warned the infrastructure could be repurposed for social surveillance. \u201c[T]his also means that technical capabilities have already been, or are being developed to monitor calls, creation, writing texts or documents, for example in search of illegal, harmful, hateful, or otherwise undesirable or iniquitous content \u2014 with respect to someone\u2019s standards,\u201d he wrote.<\/p>\n \u201cGoing further, such a model could, for example, display a warning. Or block the ability to continue,\u201d Olejnik continued with emphasis. \u201cOr report it somewhere. Technological modulation of social behaviour, or the like. This is a major threat to privacy, but also to a range of basic values and freedoms. The capabilities are already there.\u201d<\/p>\n Fleshing out his concerns further, Olejnik told TechCrunch: \u201cI haven\u2019t seen the technical details but Google assures that the detection would be done on-device. This is great for user privacy. However, there\u2019s much more at stake than privacy. This highlights how AI\/LLMs inbuilt into software and operating systems may be turned to detect or control for various forms of human activity. <\/p>\n This highlights how AI\/LLMs inbuilt into software and operating systems may be turned to detect or control for various forms of human activity. <\/p>\n Lukasz Olejnik<\/cite><\/p><\/blockquote>\n<\/figure>\n \u201cSo far it\u2019s fortunately for the better. But what\u2019s ahead if the technical capability exists, and is built in? Such powerful features signal potential future risks related to the ability of using AI to control the behavior of societies at a scale or selectively. That\u2019s probably among the most dangerous information technology capabilities ever being developed. And we\u2019re nearing that point. How do we govern this? Are we going too far?\u201d<\/p>\n Michael Veale, an associate professor in technology law at UCL, also raised the chilling spectre of function-creep flowing from Google\u2019s conversation-scanning AI \u2014 warning in a reaction post on X<\/a> that it \u201csets up infrastructure for on-device client side scanning for more purposes than this, which regulators and legislators will desire to abuse.\u201d<\/p>\n Privacy experts in Europe have particular reason for concern: The European Union has had a controversial message-scanning legislative proposal on the table, since 2022<\/a>, which critics \u2014 including the bloc\u2019s own Data Protection Supervisor<\/a> \u2014 warn represents a tipping point for democratic rights in the region as it would force platforms to scan private messages by default. <\/p>\n While the current legislative proposal claims to be technology agnostic, it\u2019s widely expected that such a law would lead to platforms deploying client-side scanning in order to be able to respond to a so-called \u201cdetection order\u201d demanding they spot both known and unknown CSAM and also pick up grooming activity in real-time. <\/p>\n\n