OpenAI\u2019s new \u2014 and first! \u2014 video-generating model, Sora<\/a>, can pull off some genuinely impressive cinematographic feats. But the model\u2019s even more<\/em> capable than OpenAI initially made it out to be, at least judging by a technical paper<\/a> published this evening.<\/p>\n The paper, titled \u201cVideo generation models as world simulators,\u201d co-authored by a host of OpenAI researchers, peels back the curtains on key aspects of Sora\u2019s architecture \u2014 for instance revealing that Sora can generate videos of an arbitrary resolution and aspect ratio (up to 1080p). Per the paper, Sora\u2019s able to perform a range of image and video editing tasks, from creating looping videos to extending videos forwards or backwards in time to changing the background in an existing video.<\/p>\n But most intriguing to this writer is Sora\u2019s ability to \u201csimulate digital worlds,\u201d as the OpenAI co-authors put it. In an experiment, OpenAI set Sora loose on Minecraft and had it render the world \u2014 and its dynamics, including physics \u2014 while simultaneously controlling the player.<\/p>\n Sora controlling a player in Minecraft \u2014 and rendering the video game world as it does so. Note that the graininess was introduced by a video-to-GIF converter tool, not Sora. Image Credits:<\/strong> OpenAI<\/p>\n<\/div>\n So how\u2019s Sora able to do this? Well, as observed<\/a> by senior Nvidia researcher Jim Fan (via Quartz<\/a>), Sora\u2019s more of a \u201cdata-driven physics engine\u201d than a creative too. It\u2019s not just generating a single photo or video, but determining the physics of each object in an environment \u2014 and rendering a photo or video (or interactive 3D world, as the case may be) based on these calculations.<\/p>\n \u201cThese capabilities suggest that continued scaling of video models is a promising path towards the development of highly-capable simulators of the physical and digital world, and the objects, animals and people that live within them,\u201d the co-authors write.<\/p>\n Now, Sora\u2019s usual limitations apply in the video game domain. The model can\u2019t accurately approximate the physics of basic interactions like glass shattering. And even with interactions it\u00a0can<\/em> model, Sora\u2019s often inconsistent \u2014 for example rendering a person eating a burger but failing to render bite marks.<\/p>\n Still, if I\u2019m reading the paper correctly, it seems Sora could pave the way for more realistic \u2014 perhaps even photorealistic \u2014 procedurally generated games. That\u2019s in equal parts exciting and terrifying (consider the deepfake implications, for one) \u2014 which is probably why OpenAI\u2019s choosing to gate Sora behind a very <\/em>limited access program for now.<\/p>\n Here\u2019s hoping we learn more sooner rather than later.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div><\/div>\n![\"OpenAI](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2024\/02\/ezgif-7-32e8e27b50-1.gif\")
<\/p>\n