The "world's first" multimodal world model has arrived!
It comes from Li Fei-Fei's World Labs.
World Labs is lavish with its praise for this new-generation world model, dubbed Atlas, emphasizing that it's not just about generating interactive videos this time around.
Modeling the world, moving cameras, simulating space and time.
In other words, Atlas can generate images and video frames with pixel-level camera control and complete 3D reconstruction, and it can understand both space and time.
Li Fei-Fei herself considers Atlas a "milestone" achievement for World Labs, giving it top billing.
Atlas has opened doors to a wide range of applications, from visual effects to robotics.
Li Fei-Fei's protege and Nvidia's robotics chief, Jim Fan, also weighed in, saying this is a major step forward for the robotics field in terms of real-to-simulation technology.
Atlas is a multimodal autoregressive diffusion transformer. Its capabilities include:
Camera-control generation: From a single image, Atlas can generate images and videos with pixel-level camera control, producing up to one minute of 1440p video.
Spatial reconstruction: Atlas can reconstruct real-world scenes from one to dozens of input images. It can generate image frames from new viewpoints and output explicit 3D representations, outperforming current state-of-the-art models trained specifically for 3D reconstruction.
Spatiotemporal simulation: Atlas simultaneously models space and time from input video, can shift the viewpoint in existing videos to create dramatic visual effects, and supports real-to-sim workflows for robotics.
Image generation: Atlas can generate images and 360-degree panoramas from text, follow complex prompts, render text accurately, and produce a wide range of visual styles.
For robot simulation, feeding Atlas just a few photos enables it to generate realistic RGB and depth data. This allows robots to be trained and tested in a wider variety of simulated environments.
A multimodal world model trained from scratch.
Technically, Atlas is an omni model.
/rIt aims to handle multiple tasks, various inputs, and outputs in a unified model architecture.
Meanwhile, spatial control is at the core of the entire model.
To achieve these goals, Fei-Fei Li's team designed a brand-new foundational infrastructure to serve as the basis for future world models: the multimodal autoregressive diffusion Transformer.
Text, images, videos, 3D data... all types of input will be anchored in 3D space to form a spatial context.
Then, Atlas generates multimodal output based on this context.
This way, for example, by putting two unrelated reference images in the same context and specifying their locations in 3D space, Atlas can stitch them together into a spatially natural and continuous world.
More specifically, multimodal refers to Atlas's ability to natively process various different data types, including text, images, camera poses, and 3D depth maps.
Videos are represented as image sequences.
/contentassist/ cannot assist with this task. Please try again.
Autoregression refers to Atlas operating on a sequence of elements, where each element can belong to one of the previously mentioned data modalities.
Atlas generates new output each time, conditioned on the existing sequence content that came before.
This flexible design is naturally suited for a variety of different tasks.
Each task is essentially just a different type of sequence - with input at the beginning and output at the end.
Diffusion models refer to Atlas as a Rectified Flow model, which generates output through a process of denoising.
Diffusion models are particularly adept at modeling high-dimensional continuous data such as images and videos. Additionally, during inference, changing the number of denoising steps allows for a balance between speed and quality.
Meanwhile, the Transformer needs no introduction, having proven to be a robust foundational architecture in the realm of world models.
/contentassist/Atlas effectively combines a plethora of ideas from large language models and video models.
Atlas can leverage numerous technologies that originally served LLM inference and acceleration, including KV Cache, cache-aware routing, and decoupled services, among others.
On the other hand, it is a latent space diffusion model, similar to modern image and video generation models, which can utilize algorithms such as diffusion distillation, classifier-free guidance, and shifted noise scheduling, and introduces more advanced VAE architectures.
The research team conducted quantitative evaluations of this new generation of world models on two key tasks: camera control generation and 3D reconstruction.
In camera control generation, the results show that Atlas outperforms state-of-the-art video models.
Moreover, the more complex the camera trajectory, the greater Atlas's advantage.
In the task of 3D reconstruction from sparse input views, Atlas also surpasses the best-performing open-source 3D reconstruction models (lower scores indicate better performance).
Notably, Atlas was designed with scaling in mind from the outset.
Li Fei-Fei's team said they have seen very favorable evidence that Atlas's capabilities will continue to improve as its scale expands.
Atlas is currently open to early access for some partner companies.
Access can also be applied for on the official website.
A key step in embodied intelligence, Real-to-Sim: Li Fei-Fei's new world model has been released, and the focus of attention is still on embodied intelligence.
Connecting the dots of World Labs' actions this year, the significance of Atlas is even clearer.
In June this year, Li Fei-Fei personally defined the world model, categorizing it into renderers, simulators, and planners.
And explicitly stated that 'the most critical is the simulator'.
Because the simulator undertakes the geometry, physics, and dynamics of the world itself, which is the foundation that rendering and action both depend on.
Immediately after, on July 21, World Labs acquired SceniX, a robotics simulation company.
On July 28, the Real-to-Sim-to-Real system was made public, emphasizing that the biggest bottleneck for robots is the lack of affordable, controllable, and scalable training experiences.
Now, Atlas has arrived.
A path has been paved from real photos and videos to 3D space, and then to robot sensor views and modifiable simulated environments.
According to World Labs' official statement, Atlas will become the underlying model for the future version of Marble and other World Labs products.
Reference link: https://www.worldlabs.ai/blog/atlas