When I visited my elderly mom in Germany recently, I realized it could be one of the last times I see her in the cozy little house she has called home for more than two decades. So I did what anyone would do: I busted out my phone and took lots of photos of the place to preserve as many memories as possible: the warm fireplace; the shelves full of familiar books; the rickety old garden bench up front that everyone signed during a special birthday celebration many years ago.<\/p>\n<\/div>\n
Then, I tried something else. I opened up Scaniverse<\/a>, a 3D scanner app from Pok\u00e9mon Go<\/em> maker Niantic, and captured some of those things as 3D objects, crouching and tiptoeing my way around them as I slowly moved my phone to record every angle and inch. The results were a bit imperfect around the edges, but they still felt profound. When I opened the scans up later, both on my phone and with a VR headset, I was able to look at that weathered garden bench from all angles, as if I was standing right in front of it. The experience touched me emotionally in ways I wasn\u2019t prepared for.<\/p>\n<\/div>\n That experience was possible thanks to Gaussian splatting, a novel method of 3D capture that was invented less than two years ago and is already taking the tech industry by storm. Both Niantic and Google are using it to build out their respective mapping products; Snap has added support for splats \u2014 which is what objects captured with Gaussian splatting are colloquially called \u2014 to its Lens Studio developer platform, and Meta wants to use Gaussian splatting to create a metaverse that looks just like the real world.<\/p>\n<\/div>\n Tech companies are enamored by Gaussian splatting for its ability to photorealistically capture, and then digitally recreate, three-dimensional objects. It could soon allow anyone to scan entire rooms and change how creatives in Hollywood and beyond record 3D video. When combined with generative AI, it has the potential not only to preserve existing spaces but also to transport us to entirely new 3D worlds.<\/p>\n<\/div>\n \u201cIt\u2019s a huge game changer,\u201d said AR \/ VR expert and investor Tipatat Chennavasin. As a cofounder and general partner of the Venture Reality Fund<\/a>, Chennavasin has a financial interest in this technology\u2019s success. As a geek and former 3D artist, he has fallen in love with it, likening it to the Star Trek<\/em> holodeck, which allowed crew members to enter holographic 3D simulations of real and imaginary spaces. \u201cWe\u2019re starting to get to a photoreal holodeck.\u201d<\/p>\n<\/div>\n Capturing objects in 3D, even on your phone, is not new. However, most prior efforts relied on polygons, the kind of triangular, cyberpunk-looking meshes you\u2019ve seen if you\u2019ve ever used a mobile AR app.<\/p>\n<\/div>\n Polygon mesh-based 3D capture and reconstruction is good enough for basic objects with flat surfaces, but it can struggle with detailed textures and complex lighting. Objects captured this way often look plasticky and unreal, and 3D-captured humans always appear to have used way too much gel rather than having individual strands of hair. \u201cIt was promising at the time, but always had huge limitations,\u201d Chennavasin said.<\/p>\n<\/div>\n All of that changed in the summer of 2023, when a group of European scientists published a paper on something they called \u201c3D Gaussian splatting.\u201d Their approach to the problem was to ditch the meshes and instead capture 3D objects as a collection of fuzzy, translucent blobs, also known as Gaussians.<\/p>\n<\/div>\n Each of these blobs is captured with exact information on its color, location, scale, rotation, and level of transparency \u2014 and when you combine millions of them, you get a much more detailed picture of a 3D object that also details how it looks from any given angle, thanks to all of this additional data. Using machine learning, they were able to capture objects with a lot more detail, in higher fidelity, and render them in real time without the need for heavy graphics-rendering rigs.<\/p>\n<\/div>\n Experts in the field were immediately blown away by the results. \u201cWe finally have the chance to have true 3D that\u2019s photo-real,\u201d Chennavasin said. \u201cIt\u2019s the JPEG moment for spatial computing.\u201d<\/p>\n<\/div>\n Niantic SVP of engineering Brian McClendon believes that Gaussian splats are the most profound advancement in the field of 3D graphics in more than 30 years. \u201cWe see it as a fundamental change,\u201d he said.\u00a0<\/p>\n<\/div>\n \u201cWe see it as a fundamental change.\u201d<\/p>\n<\/div>\n<\/div>\n According to McClendon, Gaussian splatting is going to democratize 3D capture \u2014 and Niantic wants to be at the forefront of this change. After acquiring the Scaniverse app in 2021<\/a>, Niantic added Gaussian splatting as a capture technology last year<\/a>. In August, it launched a new version<\/a> of Scaniverse that puts splatting front and center. In October, the company open sourced<\/a> its own file format for splats. And in December, Scaniverse expanded to VR, enabling users to look at Gaussian splats in Meta\u2019s Quest headsets<\/a>.<\/p>\n<\/div>\n Niantic has its own reasons for pushing splatting. Scaniverse started out as an app to capture personal memorabilia and other individual objects, but Niantic is now encouraging people to also scan statues, fountains, and other public points of interest. The company sees these scans as key components of the 3D map of the world it is building<\/a> \u2014 the same map that powers Pok\u00e9mon Go<\/em>, Peridot<\/em>, and future geospatial AR games and experiences. \u201cWe are very focused on the map, and scanning and reconstructing the outdoors,\u201d McClendon said.<\/p>\n<\/div>\n \u201cWe already have hundreds of thousands of these [types of scans] in Scaniverse right now,\u201d McClendon said. \u201cHopefully, we\u2019ll get to a million soon.\u201d<\/p>\n<\/div>\n Gaussian splats aren\u2019t just for capturing static content. Computer vision startup Gracia AI<\/a> has been using the technology to record volumetric 3D videos, which can be viewed on Meta Quest headsets. One of those clips shows a chef<\/a> preparing a meal, with viewers being able to look at the action from all angles in VR and even zoom in to observe his knife slicing through a glistening piece of raw salmon.\u00a0<\/p>\n<\/div>\n Gracia recorded this video in a professional 3D capture studio, using an array of 40 cameras pointed at the chef from all angles. That\u2019s how professionals have been recording holographic content for AR and VR experiences for years \u2014 but once again, the transition from polygons to Gaussian splats makes all the difference.<\/p>\n<\/div>\n Previously, 3D video capture presented a series of visual challenges that led to strict dress codes for captured individuals: no busy patterns, nothing translucent, nothing loose and dangling that could result in weird artifacts. When Microsoft captured David Attenborough this way several years ago, it even had to glue his collar to his shirt<\/a> and use obscene amounts of hairspray to literally avoid any loose ends that could mess up the capture process.<\/p>\n<\/div>\n \u201cIt\u2019s amazing how much creative flexibility you get with Gaussian splats.\u201d<\/p>\n<\/div>\n<\/div>\n With Gaussian splats, all of those limitations are gone. \u201cThere are no restrictions with clothing, there are no restrictions with hair,\u201d said Gracia cofounder and CEO Georgii Vysotskii, who counts Chennavasin\u2019s Venture Reality Fund among his company\u2019s investors. While previous-generation volumetric video capture required blinding amounts of light to eliminate any shadows, Gracia has been able to record scenes in almost complete darkness. \u201cYou can leave all the shadows, and use artistic lighting,\u201d Vysotskii said. \u201cIt\u2019s amazing how much creative flexibility you get with Gaussian splats.\u201d<\/p>\n<\/div>\n That\u2019s not to say there aren\u2019t still challenges. At the moment, Gaussian splatting clips still require 9GB of data per minute of video \u2014 too much for streaming or really anything beyond a short tech demo. Vysotskii said that the company is now working on reducing it to 2\u20133GB per minute, and 180-degree volumetric VR videos could require as little as 1GB of data per minute. He envisions these types of clips eventually replacing the recordings of instructors in VR workout apps like Supernatural or professional educational content because they allow users to look at instructions from all angles.<\/p>\n<\/div>\nBuilding a 3D map of the world, one splat at a time<\/h3>\n<\/p>\n
Splats are changing 3D video capture<\/h3>\n<\/p>\n
Meta\u2019s ambitious plans for Gaussian splats<\/h3>\n<\/p>\n