/technology/global-first multimodal world model has arrived!
It comes from Fei-Fei Li's World Labs.
World Labs is sparing no superlatives in describing the new-generation world model, dubbed Atlas, emphasizing that this time, it's not just about generating interactive videos:
Modeling the world, moving cameras, simulating space and time.
In other words, Atlas can generate images and video frames with pixel-level camera control and complete 3D reconstruction, and it understands both space and time.
Fei-Fei Li herself considers Atlas a 'milestone achievement' for World Labs, pinning it to the top:
Atlas has opened the door to a wide range of applications, from visual effects to robotics.
Fei-Fei Li's protege and Nvidia's robotics chief, Jim Fan, also weighed in, saying this is a major step forward for the robotics field in terms of real-to-simulation technology.
/content/ Atlas is a multimodal autoregressive diffusion transformer with capabilities including:
Camera control generation: With just one image, Atlas can generate images and videos with pixel-level camera control, with output of up to 1 minute and 1440p video.
Spatial reconstruction: Atlas can reconstruct real-world scenes based on one to dozens of input images, generating new perspective image frames and explicit 3D representations, exceeding the current state-of-the-art models trained specifically for 3D reconstruction.
Spatiotemporal simulation: Atlas can model space and time based on input videos, converting observation perspectives in existing videos and creating dramatic visual effects, while supporting robot Real-to-Sim workflows.
Image generation: Atlas can generate images and 360-degree panoramas based on text, following complex prompts, accurately rendering text, and generating a large number of different visual styles.
For robot simulation, Atlas only needs to be fed a few photos to generate realistic RGB and depth data, allowing robots to be trained and tested in more diverse simulation spaces.
Doubao Unveils World Model Trained from Scratch with Multimodal Capabilities
In terms of technical details, Atlas is an omni-model.
Its goal is to handle multiple tasks, multiple inputs, and output data in a unified model architecture.
/contentassist1] Meanwhile, spatial control is at the core of the entire model.
To achieve these goals, Fei-Fei Li's team designed a brand-new foundational infrastructure to serve as the basis for future world models - the multimodal autoregressive diffusion Transformer.
Text, images, videos, 3D data... all types of input will be anchored in 3D space, forming a spatial context.
Then, Atlas generates multimodal output based on this context.
For example, by putting two unrelated reference images in the same context and specifying their locations in 3D space, Atlas can stitch them together into a spatially natural and continuous world.
More specifically, multimodal refers to Atlas's ability to natively process a variety of different data types, including text, images, camera poses, and 3D depth maps.
Videos are represented as image sequences.
Each image and depth map corresponds to a specific camera pose.
/auto-regressive, referring to Atlas operating on a sequence of elements, where each element can belong to one of the previously mentioned data modalities.
Each time Atlas generates a new output, it uses the existing sequence content as a condition.
This flexible design is naturally suitable for various different tasks.
Essentially, each task is just a different type of sequence - with input at the front and output at the back.
Diffusion model, referring to Atlas as a Rectified Flow model, which generates output through step-by-step denoising.
The diffusion model is particularly adept at modeling high-dimensional continuous data such as images and videos, and during inference, it can balance speed and quality by adjusting the number of denoising steps.
The Transformer, as is well known, has also proven to be a robust foundational architecture in the realm of world models.
In other words, Atlas actually combines a large number of ideas from big language models and video models.
Atlas can leverage various technologies that originally served LLM inference and acceleration, including KV Cache, cache-aware routing, and decoupled services, among others.
On the other hand, it is a latent space diffusion model similar to modern image and video generation models, which can utilize algorithms such as diffusion distillation, classifier-free guidance, and shifted noise scheduling, and introduce more advanced VAE architectures.
The research team conducted quantitative evaluations of this new generation of world models on two key tasks: camera control generation and 3D reconstruction.
In camera control generation, the results show that Atlas outperforms the state-of-the-art video models.
Moreover, the more complex the camera trajectory, the greater the advantage of Atlas.
In the task of 3D reconstruction from sparse input views, Atlas also surpassed the best-performing open-source 3D reconstruction models, with lower scores indicating better performance.
Notably, Atlas was designed with scaling in mind from the outset.
The team led by Li Fei-Fei stated that they have seen very favorable evidence that Atlas's capabilities will continue to improve with scale.
Atlas is currently offering early access to select partners.
Access can also be requested through the official website.
A key step in embodied intelligence, Real-to-Sim, has drawn attention to the field, particularly with the release of Fei-Fei Li's new world model.
Considering World Labs' actions this year, the significance of Atlas has become clearer.
In June, Fei-Fei Li defined world models, categorizing them as renderers, simulators, and planners.
She emphasized that the simulator is the most critical component.
This is because the simulator handles the geometry, physics, and dynamics of the world itself, serving as the foundation for both rendering and action.
Shortly after, on July 21, World Labs acquired SceniX, a robotics simulation company.
On July 28, the Real-to-Sim-to-Real system was made public, highlighting that the biggest bottleneck for robots is the lack of affordable, controllable, and scalable training experiences.
Atlas has arrived.
A path has been paved from real photos and videos to 3D space, and then to robot sensor views and modifiable simulated environments.
According to World Labs, Atlas will become the underlying model for future versions of Marble and other World Labs products.
Reference link: https://www.worldlabs.ai/blog/atlas