Two months ago, we introduced Atlas, an autoregressive world model that generates frames one at a time with pixel-perfect camera control.
Today, we are releasing a research preview allowing you to explore worlds in real-time with Atlas using Frameboy, a demo that streams frames from cloud GPUs to your browser at up to 15 fps as you control the camera. Frameboy serves a variant of Atlas that has been highly optimized to achieve the fastest, most efficient inference possible while maintaining excellent generative novel view synthesis capabilities. We have also integrated this real-time functionality into Viewfinder, a web app that lets you build up your own scenes frame-by-frame with Atlas, then explore them live with a streaming Frameboy overlay.
Why speed matters
Atlas is a world model that can be used to explore an imaginary 3D world defined by a single image or to navigate a real world environment captured from a handful of different viewpoints. With Atlas, you can explore a famous landmark, scout out virtual viewpoints for a film shoot, inspect progress on a construction site, flesh out a concept for a 3D videogame level, or simulate an environment for a robot to navigate through.
One key factor that dramatically influences how users interact with Atlas is speed. If users have to wait tens of seconds or even multiple minutes to see a video generated by Atlas, it forces them to be cautious and methodical in building out scenes and planning camera paths, limiting their iteration cycles and impacting their ability to hone in on their desired result.
In contrast, if Atlas can generate outputs at a more interactive rate, users can react to adjust their input quickly and iteratively. When the cost of a “bad idea” is near zero, there are no bad ideas – what was formerly a costly mistake or error simply becomes exploration.
Two views of the same living room. Manual iteration: a user clicks four keypoint cameras into the room, presses generate and waits before the rendered video arrives, then drags a keypoint and waits again, twice more. Real-time interaction: a user drives a camera round the room and its view updates continuously.
Manual iteration with a slow generative model requires cautious and thoughtful prompting, incurring high cognitive load. In contrast, interacting with a real-time model allows for quick, intuitive exploration.
Once latency and cost drop low enough, users no longer even perceive a discrete act of “generation” – the model can simply stream frames back continuously as they navigate through the scene. After building initial prototypes around real-time variants of Atlas, we realized that interacting with the model in this way results in a completely different experience than triggering frame generation manually.
Frameboy: Atlas in real-time
Last year, we released a research preview for RTFM, an efficient and lightweight frame generation model. Atlas takes concepts from RTFM and scales them up, so we were inspired to revisit and remake a new demo with this familiar form factor, bigger and better than before.
Compared to RTFM, Atlas is a much larger model trained on orders of magnitude more data. Where RTFM was primarily designed for interpolating between a preexisting set of viewpoints, Atlas is capable of generating scenes from scratch, using as little context as a single input image or text caption.
Atlas also retains and improves upon RTFM’s capability for novel view synthesis from a collection of images. This means you can use the Frameboy demo to navigate around real world scenes, such as these famous landmarks.
Viewfinder: build and explore your own worlds
We envision a future where models like Atlas can serve as powerful generative renderers to interactively build and visualize rich, complex 3D worlds from intuitive, easy to specify inputs, like a collection of images. To demonstrate this vision, we have also integrated this real-time capability as a streaming overlay render mode within our web app Viewfinder wrapping Atlas.
(Viewfinder is currently available to test in closed beta; you can sign up to our waitlist here.)
Frameboy lets users explore a curated list of scenes, but Viewfinder goes a step further: it allows the user to build out a scene frame-by-frame from their own inputs, navigating around and generating 3D-consistent views of their world from different camera angles as if using a virtual camera. Each new frame is generated by Atlas, including both color and depth to help visualize the 3D space being created. Once the user is ready, they can trigger the Frameboy overlay to stream generated frames as they explore.
Frameboy currently must run on a powerful GPU server in the cloud and is not cheap compared to local rendering methods. However, if history is any indication, what starts slow and expensive becomes fast and cheap as both hardware and software evolve to meet the demands of new algorithms. We are excited to continue exploring the potential of this technology and pushing the limits of what is possible with future iterations of our models.
