Author: Li Fei-Fei
Compiled by: Jia Yang
The concept of a "world model" has been the hottest and most confusing term in AI since 2025. When Sora was released, OpenAI called it a world simulator; Genie lets you walk around in generated scenes and also calls it a world model; robotics companies claim they’re building world models; NVIDIA says Omniverse is the infrastructure for world models; even game engines have been pulled into this narrative. Everyone is using the same word, but they’re all referring to completely different things.
Today, Li Fei-Fei published a new article on her personal Substack, clarifying this concept. She first revisits the most classic diagram from reinforcement learning textbooks (the POMDP loop: agent → action → state → observation → agent), then points out that what is now called a “world model” is actually three different projections of this loop. The renderer outputs pixels (observations), the simulator outputs states, and the planner outputs actions. The classification is straightforward: it depends on which part of the loop is being output.
(Source: MIT Technology Review)
She concludes that among the three, the renderer is the most commercially mature but has a ceiling (being visually appealing does not equate to physical accuracy), the planner is the most exciting but furthest from real-world deployment (the gap between lab demonstrations and practical usability remains vast), while the simulator is a severely underestimated key hub. Because simulators operate at the levels of geometry, physics, and dynamics, they can project upward into pixels for human consumption and downward into action outcomes for robots. Mastering simulation means possessing the foundation for both rendering and planning; the reverse is not true.
This article is also World Labs’ product manifesto. Their Marble is already outputting both Gaussian splatting and collision meshes, attempting to unify the renderer and simulator into a single model. The conclusion of the article envisions a unified foundational world model capable of freely switching between rendering, simulation, and planning based on downstream needs. Whether this vision can be realized remains to be seen, but as an analytical framework, the tripartite distinction between renderer/simulator/planner may indeed help cut through some of the noise surrounding the current “world model” concept.
Translate the entire text as follows.
“The world is the totality of facts, not of things.” — Wittgenstein, Tractatus Logico-Philosophicus, 1921
The world is not made of words.
In a previous article, we proposed that spatial intelligence is the next frontier of AI, and that world models are the pathway to it. Here, the World Labs team and I want to go one level deeper: among the many things now labeled as “world models,” which functional components truly constitute this capability, and what are their respective purposes?
Language models give machines powerful control over concepts, words, and reasoning, but the physical world—whether virtual or real—operates on an entirely different foundation. Language models learn the statistical structure of text, while world models learn the statistical structure of space and time: how light falls on a surface, what a garden looks like from an angle never captured by a camera, and how objects respond to forces and obey the laws of physics.
This has made “world model” one of the most important—and most overused—terms in AI today. Computer vision, robotics, reinforcement learning, and generative AI all claim to be building world models, but each refers to something entirely different. A video model that generates beautiful but physically impossible flames, a language model that improvises playable games, and a physics engine that faithfully simulates combustion are all called by the same name.
The ancient Greeks never agreed on what the world was made of—whether fire, water, or indivisible atoms—because “the world” was never a single thing; it was always a stand-in term used by a philosopher to reason about some totality. AI inherits the same problem, and it happens to occur at the very moment when precision in this field is most needed.
The closed loop behind the taxonomy
To clarify this confusion, start with a diagram older than all the technologies mentioned above. All reinforcement learning textbooks, including the classic Sutton and Barto, have used variations of the same diagram for decades to illustrate how agents interact with the world. The formal name of this diagram is the Partially Observable Markov Decision Process (POMDP), and the term “world model” was originally defined within this tradition.
An agent (which can be a person, a robot, or a software system) performs actions. These actions change the state of the world. But the agent can never directly observe the state itself—it receives only observations: photons striking the retina, sensor readings, pixels in video frames. New observations guide new actions, in a continuous cycle.
The term "state" needs to be broken down, as its meaning shifts across different fields. This is not the state as chemists define it—no distinction between solid, liquid, and gas. Here, state refers to the state as understood by physicists and roboticists: a complete description of everything happening in the world at a given moment, including every object, its position, its velocity, and every property. The state is the underlying reality of the world—principally complete, yet forever unobservable directly by any agent within it. Observation is the agent’s partial perspective on this reality. Action is the agent’s response based on that perspective.
This closed loop (agent → action → state → observation → agent) is precisely the structure that gives the term "world model" its technical meaning. The phrase itself is older, dating back to Kenneth Craik’s 1943 proposal that the mind reasons by running "small-scale models" of reality, and it was introduced into the neural network field in the late 1980s and early 1990s. This same closed loop also explains what the term means when used today. The various things now called world models are in fact different projections of this same loop, each outputting a different component of the loop.
Three functions of the world model
The first world model is a renderer. A renderer outputs observations—specifically, pixels oriented toward the human eye—with the most important quality metric being visual fidelity. A video model that transforms a text prompt into a cinematic aerial shot is a renderer; interactive systems like Google’s Genie 3 or World Labs’ own RTFM are also renderers, generating frames in real time based on user input. These models lack explicit understanding of three-dimensional structure. They generate what an observer would see, not what things actually look like. The buildings in an aerial shot may appear flawless from above, but try navigating through the city below, and they would collapse.
The second is a simulator. A simulator outputs a state: a geometrically, physically, and dynamically faithful representation of the world, on which both humans and computer programs can perform computations and interact. While a renderer’s contract is purely visual, a simulator’s contract is structural—it requires geometry to be physically valid, physics to obey Newton’s laws, and dynamics to behave as expected under physical principles. Simulators serve two types of users. Professionals such as architects, designers, filmmakers, and game developers require accuracy beyond visual plausibility. Computer programs such as reinforcement learning agents, robot controllers, and autonomous vehicles use simulators as training grounds to interact with the world at scale, testing scenarios that would be too dangerous, expensive, or impossible to execute in reality.
The third is the planner. The planner outputs actions. Given an observation and a goal, the planner answers the question: What should the agent do next? In many ways, the planner is the inverse of the renderer. The renderer takes actions as input and produces observations, while the planner takes observations as input and produces actions, thereby closing the perception-action loop. Vision-language-action models (VLA), model-based systems, and the new wave of World Action Models are all different attempts at planners: enabling systems to determine what a robot should do in an unstructured world.
These three categories cover most of the practical work being implemented today, and distinguishing between them is useful in practice. However, these categories are not fundamentally separate; they share the same underlying knowledge about how the world works: geometry, physics, and dynamics. A model capable of rendering a cup from any angle should, in principle, also be able to simulate what happens when the cup is pushed and plan how a hand might pick it up. An increasing number of the most interesting research efforts are intentionally blurring the boundaries between these three areas.
Image | Three World Models (Source: Substack)
Why simulation is the critical hub
Among the three categories, the simulator has received the least public attention, yet it is the most important of the three. This article aims to correct this imbalance.
Renderers are currently the most commercialized. Numerous image-to-video and text-to-video products are rapidly expanding in both consumer and enterprise markets. Google’s Nano Banana model has brought renderer-level image generation capabilities to potentially hundreds of millions of users. The technology is real, and the market is real. However, the goal of renderer optimization is visual plausibility rather than physical accuracy—a crucial ceiling. Their outputs are beautiful, but you cannot use them to design a building or train a robot.
The planner is the most exciting and least mature component, closely tied to the rapidly evolving field of robot learning. Over the past two years, this field has produced numerous robotic demonstrations that look impressive in videos, but we must be honest about what these demonstrations actually show. Almost all of them are confined to highly restricted laboratory environments with limited object varieties and short task durations. None have been validated for the complexity, diversity, and duration required for real-world deployment. The gap between a compelling demo video and a robot that can reliably operate in a kitchen, warehouse, or operating room remains enormous.
Nevertheless, the scale of commercial bets remains substantial. A wave of well-funded new entrants is racing to launch general planning systems, while major infrastructure players are building planning capabilities on top of broader simulation stacks.
Simulation is the bridge connecting the two. If language is an abstraction of the world and pixels are a projection of the world, then geometry, physics, and dynamics are the world itself. The simulator must operate at this level: it is the structural skeleton from which both visual representations (for the renderer) and action consequences (for the planner) can be derived.
A model that has mastered simulation can project its understanding as pixels for human consumption and as action predictions for embodied agents. A model that has mastered only rendering or only planning cannot do either. The commercial opportunity here is enormous. Just NVIDIA’s Omniverse, according to the company’s estimates, targets a market size exceeding a trillion dollars, encompassing factories, warehouses, supply chains, and digital twins. Robot training, autonomous vehicle testing, architectural visualization, engineering design, and drug discovery all rely on some form of simulation.
The most challenging open problems in this field are also concentrated here. Three-dimensional data with explicit geometry, material properties, and physical annotations is orders of magnitude scarcer than internet videos used for renderer training. The sim-to-real gap—the difference between how objects behave in simulation versus the real world—still persists. Generative simulators introduce additional risks: AI-generated geometry may appear correct but contain self-intersections or incorrect proportions, leading to absurd results in physical simulations. The computational cost of large-scale multi-physics simulations (rigid bodies, deformable objects, fluids, and fabrics all interacting simultaneously) remains several orders of magnitude higher than simulations in a single domain.
At World Labs, Marble is our first step in this direction. It accepts multimodal inputs (text, images, video, or spatial sketches), generates explorable 3D environments, and outputs Gaussian splats for visual exploration and collision meshes for physics engines. But Marble is only the first chapter of a much longer arc. As the boundaries between rendering, simulation, and planning begin to dissolve, the entire field is writing this story.
The boundaries are dissolving, and what happens next
The most important trend in this field right now is that three categories are beginning to converge. The underlying consensus is that the knowledge required to render a world, simulate it, and act within it is largely the same. Using the previous example, a model that truly understands how a cup sits on a table—its geometry, material properties, response to forces, etc.—should be able to render the cup from any angle, simulate what happens when the cup is pushed, and plan a hand motion to pick it up. The three categories are three projections of the same underlying understanding.
For instance, recent work—though still limited in volume but growing—from various robotics labs has demonstrated a possibility that holds at least conceptually: a pre-trained video renderer can serve as a backbone network for jointly predicting the world and actions, enabling a single model to simultaneously imagine “what will happen” and “what to do,” thereby bridging the gap between renderer and planner. World Labs’ Marble already outputs both Gaussian splats and collision meshes from a single model, dissolving the boundary between renderer and simulator. At every level, systems are shifting from passive outputs to interactive ones: renderers are becoming responsive to action conditions, simulators are generating worlds that are more controllable and editable, and planners are beginning to engage in deliberate reasoning rather than merely reacting.
The logical endpoint is a unified world model: a foundational model capable of rendering photorealistic views, generating physically accurate structures, planning sequences of actions, and switching between different output modalities based on the needs of downstream users. We will still face a series of significant challenges. The data landscape is highly imbalanced: renderers have access to vast amounts of internet video, while simulators and planners suffer from severe shortages of 3D assets and robotic demonstration data. Optimization for visual aesthetics may come at the cost of the precision required for robotics or high-fidelity simulation. Reconciling these tensions within a single architecture is the central open problem in today’s world model research—and what World Labs is committed to solving as it continues to evolve Marble.
(Source: Substack)
But the overall direction is clear. Since the late 1980s, this field has consistently bet on the same idea: that if a world model is rich enough, everything an agent needs to perceive, construct, and act within the world is already contained within it. This bet is now driving the research of an entire generation. What has truly tipped the scales is the ongoing convergence: rendering, simulation, and planning—each of which has already grown into multi-billion-dollar industries—were once separate research paths, but are now beginning to merge. As these boundaries dissolve, their convergence will redefine something larger: the relationship between machine intelligence and the physical world it inhabits—the long-term trajectory of spatial intelligence.
Language gives machines a way to talk about the world. World models are the means by which machines ultimately understand, imagine, reason, and interact with it.
Reference: 1. https://drfeifei.substack.com/p/a-functional-taxonomy-of-world-models