70 systems · 36 organisations · last checked 10 Oct 2026

World models

A world model is an AI model that predicts how a scene will change, often in response to an action. This page explains how world models developed, the main types, who builds them and where they are used.

What a world model is

A world model takes what a camera sees now, and often an action, and predicts what the camera will see next.

A robot can practise a task inside a world model instead of in the real world.
A world model can test a robot’s policy before the policy runs on a real robot.
A world model can generate videos and scenes that are used as training data.
A world model is learned from data, such as videos and robot recordings. A physics simulator is written by engineers from the equations of physics.

How world models developed

The chart shows 70 systems released between 2015 and 2026. Each row is one type of world model. A line joins a system to an earlier system that it builds on.

Used for

Weights are publicWeights are not publicNot knownBuilds on an earlier systemLink inferred by usYears with more releases are drawn wider. Select a system to see its details and sources.

2015–2017
Predicting what happens next
Researchers trained networks to predict game frames or robot camera images from actions, and to predict object physics with graph networks.
2018–2022
Learning inside a world model
Agents such as World Models, PlaNet, Dreamer, MuZero and TD-MPC learned compact models of their environment and planned or trained inside them.
2023–2024
Video models as simulators
Large video generators such as UniSim, GAIA-1, Sora and Genie were trained on internet or driving video and presented as simulators of the world.
2025–2025
Foundation and real-time world models
Companies released open world foundation models (Cosmos, Wan) and real-time interactive worlds (Genie 3, Matrix-Game 2.0), and robot teams used them for data and evaluation.
2026–2026
World models as robot policies and test tracks
World models were used directly as robot policies (DreamZero, Cosmos Policy, 1XWM) and as closed-loop simulators for robots and cars (GAIA-4, Oasis 3, Waymo World Model).

Types of world models

There are 8 main types. They differ in what they take in, what they predict and what they are used for.

Latent dynamics models

A latent dynamics model compresses each observation, such as a camera image, into a short list of numbers and learns to predict how that list changes after each action. It takes past observations and actions as input and outputs the predicted next state, usually together with the reward the agent will receive. It is used to plan actions or to train a decision-making program (a policy) on predicted experience, so the agent needs fewer real trials.⁵⁶⁷⁺⁶ Inferred

How it works

The agent acts in a game, a simulator or the real world and records what it observed, which action it took and what reward it received.
An encoder network compresses each observation into a short list of numbers called the latent state.
A dynamics network learns to predict the next latent state and the reward from the current latent state and the chosen action.
The agent plans, or trains its policy, by running many predicted futures inside the model instead of in the real environment.
The agent then acts with the improved policy, records new data and updates the model, and the cycle repeats.

Takes actions

Yes

Used for

planningtraining policiesrobot controlgames

Strengths

Predicting a short list of numbers is cheaper than predicting full images, so a planner can test many candidate actions quickly. The 2026 robotic manipulation survey (arXiv 2606.00113) gives this as the main appeal of latent models.
Learning needs fewer real trials because most practice happens inside the model. DayDreamer reports a four-legged robot learning to roll off its back, stand up and walk from scratch in 1 hour, without resets.
One method can cover many tasks with one setting. DreamerV3 reports results on over 150 diverse tasks with a single configuration, and TD-MPC2 reports a single 317M parameter agent performing 80 tasks.
The approach works without being told the rules. MuZero learned Go, chess, shogi and 57 Atari games without knowledge of their underlying dynamics.

Limits

Errors add up when the model predicts many steps ahead, because each step starts from the previous prediction. The 2025 embodied AI survey (arXiv 2510.16732) names this error accumulation as the main weakness of step-by-step prediction.
Compressing an image into a short list of numbers can drop small visual details that matter for the task. The DIAMOND authors make this point about compact discrete latent states.
People cannot easily read the internal numbers, so it is hard to check whether a prediction is physically plausible. The 2026 robotic manipulation survey calls this a loss of auditability.
A model trained for one task or one robot often transfers poorly to others. The 2025 embodied AI survey and the 2026 robotic manipulation survey both report this for task-specific world models.

Examples

World ModelsGoogle Brain; NNAISENSE; Swiss AI Lab IDSIA (USI & SUPSI) · 2018
PlaNetGoogle Brain; DeepMind; Google Research; University of Toronto; University of Michigan · 2018
MuZeroDeepMind · 2019
DayDreamerUniversity of California, Berkeley · 2022
DreamerV3Google DeepMind; University of Toronto · 2023
TD-MPC2University of California San Diego · 2023

Predictive representation models

A predictive representation model hides part of an image or video and learns to predict a numeric description (features) of the hidden or future part, without drawing any pixels. It takes images or video, sometimes with robot actions, and outputs predicted features. It is used as a general model of what video shows, and with actions added it is used to plan robot movements toward a goal image.^1094159+1 Inferred

How it works

An encoder network turns images or video frames into features.
Parts of the input are hidden, or lie in the future, and a predictor network estimates the features of those parts from the visible parts.
Training compares the estimate with the encoder's features of the real hidden parts and adjusts both networks. Extra rules stop the networks from giving every input the same features, a failure called collapse.
For robot control, an action-conditioned predictor is trained on robot video. V-JEPA 2-AC used less than 62 hours of unlabeled robot videos from the Droid dataset.
To plan, the system tries candidate action sequences, predicts the resulting features and picks the sequence whose prediction is closest to the features of a goal image.

Takes actions

Optional

Used for

pretraining visual modelsplanningrobot control

Strengths

No computing is spent on drawing texture and lighting. Meta's V-JEPA 2 and NYU's DINO-WM both predict features without reconstructing pixels.
Most training uses video without action labels. V-JEPA 2 was pretrained on over 1 million hours of internet video and then needed less than 62 hours of robot video for its action-conditioned part.
Robots can be controlled in new places without new data. Meta deployed V-JEPA 2-AC zero-shot on Franka arms in two different labs for picking and placing objects, without collecting data from the robots in these environments.
Small models can plan quickly. The LeWorldModel authors report about 15M parameters, training on a single GPU in a few hours, and planning up to 48x faster than foundation-model-based world models.

Limits

The predictions are lists of numbers, so people cannot watch them to check whether they are right.
Long plans are hard. The V-JEPA 2 authors report that prediction accuracy falls in longer rollouts and that the number of possible action sequences grows exponentially with the planning horizon.
Results depend on the physical setup. V-JEPA 2-AC was sensitive to camera position, and its authors tried camera positions by hand before finding one that worked.
Goals are usually given as images. The V-JEPA 2 authors note that language would be a more natural way to state goals for robots in everyday settings.
Training can fail through collapse. The LeWorldModel authors describe earlier methods as fragile and dependent on extra losses, moving-average tricks, pretrained encoders or extra supervision to avoid it.

Examples

I-JEPAMeta AI (FAIR), with McGill University, Mila and New York University · 2023
DINO-WMNew York University (Courant Institute); Meta AI · 2024
V-JEPA 2 and V-JEPA 2-ACFAIR at Meta, with Mila and Polytechnique Montréal · 2025
LeWorldModelMila & Université de Montréal; New York University; Samsung SAIL; Brown University · 2026

Action-conditioned video models

An action-conditioned video model predicts the next video frames of a scene given the actions that a robot, a car or a game agent will take. It takes recent camera frames and a sequence of planned actions as input and outputs the video those actions would produce. Teams use it to test robot and driving software without the real machine, to create training data and to train policies inside the predictions.^212537+4 Inferred

How it works

Collect videos from robots, cars or games together with the action taken at each moment, such as arm positions, steering or game controls.
Start from a video model, often one pretrained on general video, and train it to predict the next frames from past frames plus the recorded actions.
To test a policy, feed the policy's actions into the model, show the policy the predicted frames, and repeat step by step.
Judge whether the predicted video shows the task succeeding, and compare policies by their predicted success.
Successful predicted runs can also be used as extra training examples for the policy.

Takes actions

Yes

Used for

evaluating policiestraining policiesgenerating training datatesting self-driving software

Strengths

Policies can be compared without running real robots. The Ctrl-World authors report that their model can accurately rank policy performance without real-world robot rollouts, and Google DeepMind checked its Veo-based evaluator against 1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks.
Rare or unsafe situations can be produced on request. Wayve built GAIA-2 to simulate both common and rare driving scenarios, and the Veo-based evaluator edits scenes to add new objects, backgrounds and distractors.
Predicted runs can improve a policy. Ctrl-World reports that fine-tuning on successful trajectories generated in the model improved policy success by 44.7%.
Policies trained inside the model can work on real robots. The UniSim authors report vision-language and reinforcement learning policies that were deployed in the real world zero-shot after training purely in their simulator.

Limits

The model can show events that cannot happen. The UniSim authors report hallucinations when an action does not fit the scene, and 1X reports many generations that fail to adhere to physical laws.
Objects can change shape or colour or disappear during interaction, as 1X reports for its world model.
Memory is short. The UniSim authors give the example of an apple that disappears from a drawer when the frames showing it being put there are no longer in the model's input.
Accuracy drops for robots, scenes and tasks that are missing from the training data, which the UniSim authors list as limited out-of-domain generalization.
A video that looks right can still show a physically wrong result for the robot. The 2026 robotic manipulation survey states that visual plausibility is not equivalent to action validity.

Examples

GAIA-1Wayve · 2023
UniSim (Learning Interactive Real-World Simulators)UC Berkeley; Google DeepMind; MIT; University of Alberta · 2023
1X World Model1X Technologies · 2024
GAIA-2Wayve · 2025
Ctrl-WorldStanford University; Tsinghua University · 2025
Veo world simulator for Gemini Robotics policy evaluationGoogle DeepMind (Gemini Robotics Team) · 2025

World-action models

A world-action model predicts what the scene will look like next and also outputs the actions a robot should take, in one model. It takes camera frames, a task instruction and the robot's state, and outputs robot actions and, in the examples here, predicted future frames as well. It is used directly as the robot's controller.^19111112+4 Inferred

How it works

Start from a model pretrained to generate video. GR-1 uses large-scale video generative pretraining, and DreamZero builds on a pretrained video diffusion backbone.
Train it on robot data so it predicts future frames and robot actions together.
At run time the robot feeds in its camera view and instruction, and the model outputs the next actions.
Some designs skip drawing the video at run time to save time. UVA decodes video and actions separately so that action output stays fast.
An earlier design generates a video plan first and then extracts actions from it with a separate model. UniPi works this way.

Takes actions

Yes

Used for

robot control

Strengths

NVIDIA reports that DreamZero generalised to new tasks and environments over 2x better than the vision-language-action models it was compared with, in real robot experiments.
Video of other robots or of humans can help. DreamZero reports a relative improvement of over 42% on unseen tasks from 10-20 minutes of video-only demonstrations.
ByteDance Research reports that GR-1 raised the success rate on the CALVIN benchmark from 88.9% to 94.9%.

Limits

Large models are slow. DreamZero needed model and system optimizations to run a 14B model for real-time closed-loop control at 7Hz.
Very fine precision is hard. The DreamZero authors note limitations on tasks requiring sub-centimeter precision, such as key insertion.
The UVA authors state that video-generation-based methods have struggled to match direct policy learning in action accuracy and inference speed.
When prediction is hidden inside the controller, it is hard to tell whether the model learned useful dynamics or only better image features, as the 2026 robotic manipulation survey notes.

Examples

UniPiMIT; Google DeepMind; UC Berkeley; Georgia Tech; University of Alberta · 2023
GR-1ByteDance Research · 2023
UVA (Unified Video Action Model)Stanford University · 2025
DreamZeroNVIDIA · 2026

Video generators

A video generator creates a new video clip from a text description, a starting image or a short video, and it does not receive step-by-step actions from a player or a robot. Its output is the video clip. It is used to make video content and as a base model that builders later fine-tune into action-conditioned world models, and OpenAI describes scaling such models as a path towards general-purpose simulators of the physical world.^1154554+1 Inferred

How it works

Builders collect very large sets of videos and images and attach a text description to each. NVIDIA reports about 20M hours of raw video for Cosmos, and OpenAI trained a captioning model to describe every video in Sora's training set.
A compression network shrinks each video into a smaller grid of numbers so the generator can process long, high-resolution clips.
The generator learns to turn random noise into a clip that matches the description, or to continue a given clip.
At use time a person supplies a prompt, a start image or a clip, and the model produces the new clip.
Builders can fine-tune the same model on robot or driving video with camera paths or actions as extra inputs, which turns it into an action-conditioned model. The Cosmos paper shows such fine-tuning for camera control, robotic manipulation and autonomous driving.

Takes actions

Used for

content creationbase model for other world modelsgenerating training data

Strengths

Training needs only videos, images and text descriptions, which exist in very large amounts. NVIDIA built Cosmos from about 20M hours of raw video.
Large models show some 3D consistency and object permanence without being designed for it. OpenAI reports that Sora keeps people and scene elements consistent as the camera moves, and often, though not always, keeps objects that are hidden or leave the frame.
One model can perform visual tasks it was not trained for. Google DeepMind reports Veo 3 segmenting objects, detecting edges, editing images and solving mazes without task-specific training.
A general model can be specialised for a robot or a car with less data. The Cosmos paper states that the dataset for post-training can be much smaller than the pretraining data.
Generated videos can become robot training data. NVIDIA's DreamGen adapts image-to-video models to a robot, recovers actions from the generated videos, and reports a humanoid robot performing 22 new behaviors while its teleoperation data covered a single pick-and-place task in one environment.

Limits

Generated clips can break physics. OpenAI states that Sora does not accurately model the physics of many basic interactions, such as glass shattering.
Long clips can drift. OpenAI lists incoherence in long samples and objects that appear spontaneously, and NVIDIA reports objects unexpectedly appearing from below in some outputs of its autoregressive Cosmos models.
The model has no input for a specific robot action, so it cannot directly show what happens if a robot moves its arm a certain way.
Uses beyond video content are often proposals. The Cosmos paper lists policy evaluation, policy training, planning and synthetic data generation as uses, and states that it does not include empirical results for them.
Builders often do not publish the model details. OpenAI's Sora report states that model and implementation details are not included.

Examples

SoraOpenAI · 2024
Cosmos world foundation models (Cosmos-Predict1)NVIDIA · 2025
DreamGenNVIDIA, with University of Washington, KAIST, UCLA, UCSD, CalTech, NTU, University of Maryland and UT Austin · 2025
Veo 3 (studied in Video models are zero-shot learners and reasoners)Google DeepMind · 2025

Real-time interactive worlds

A real-time interactive world model draws a game-like world one frame at a time while a person or an AI agent controls it with a keyboard, a mouse or text commands. It takes the latest control input and the frames it has already drawn, and it outputs the next frame fast enough for live play. It is used in games and interactive media research and as a place to train and test AI agents.^283236+3 Inferred

How it works

Collect many hours of gameplay or video. GameNGen recorded an AI agent playing DOOM, Matrix-Game 2.0 produced about 1200 hours of video from Unreal Engine and GTA5 environments, and Genie learned from unlabelled Internet videos.
Train a model to predict the next frame from recent frames and the control pressed at that moment. Genie learns its own small set of controls from videos that have no control labels.
Make each frame fast enough for live play, for example by reducing the number of generation steps per frame.
During play, each new frame joins the model's recent history, the player reacts, and the loop repeats.

Takes actions

Yes

Used for

gamescontent creationtraining policiesevaluating policies

Strengths

Worlds can be started from a prompt without hand-built 3D assets. Genie can be prompted with text, synthetic images, photographs and sketches, and Genie 3 generates worlds from a text prompt.
Live play is possible on current hardware. GameNGen runs at 20 frames per second on a single TPU, Oasis at 20 frames per second, Matrix-Game 2.0 at 25 FPS and Genie 3 at 24 frames per second at 720p.
Short clips can be hard to tell apart from the real game. The GameNGen authors report that human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation.
The worlds can host AI agents. Google DeepMind ran its SIMA agent inside Genie 3 worlds and gave it goals to pursue.

Limits

Memory is short. GameNGen has access to a little over 3 seconds of history, the first Genie model is limited to 16 frames of memory, and Oasis lists limited memory over long horizons.
Sessions last minutes. Google DeepMind states that Genie 3 supports a few minutes of continuous interaction.
Control is limited. Genie 3 lists a limited action space and difficulty simulating other agents, and Oasis lists difficulty with precise inventory control.
Speed is hard to reach. The first Genie model ran at around 1FPS, and the Matrix-Game 2.0 authors say earlier interactive models were held back by lengthy inference steps.
Access can be restricted. Genie 3 was announced as a limited research preview, and the Genie authors chose not to release model checkpoints or training data.

Examples

GenieGoogle DeepMind (one author also at University of British Columbia) · 2024
DIAMONDUniversity of Geneva; University of Edinburgh; Microsoft Research · 2024
GameNGenGoogle Research; Google DeepMind; Tel Aviv University · 2024
OasisDecart; Etched · 2024
Genie 3Google DeepMind · 2025
Matrix-Game 2.0Skywork AI · 2025

3D and 4D world models

A 3D world model builds a scene as 3D structure, such as points, surfaces or a grid of occupied space, so the scene stays the same when seen from a new angle. It takes text, images, video or recorded sensor data and outputs a 3D scene that can be explored, rendered from any viewpoint or exported to games and simulators. Some versions also predict how the 3D scene will change over the next seconds, which self-driving research uses to forecast the road ahead.^116117118+4 Inferred

How it works

Take a prompt or a real recording. Marble accepts text, images, video or coarse 3D layouts, WonderWorld starts from a single image, and Waabi's UniSim starts from one recorded drive with camera and LiDAR data.
Estimate depth and shape. Some systems first generate a 360-degree panorama, and HunyuanWorld 1.0 uses panoramic images as stand-ins for the whole world.
Store the result in a 3D format such as Gaussian splats (a large set of small semi-transparent particles), meshes, or grids that mark which spaces are occupied.
Render new views, let a user move through the scene, export it to a game engine, or edit it, for example by adding or removing cars.
In forecasting versions such as OccWorld, a second model predicts the next occupancy grid and the car's own path from the previous grids.

Takes actions

Optional

Used for

virtual worlds and 3D contentgamestesting self-driving softwarerobot simulationplanning

Strengths

The scene is consistent from every angle because it is stored in 3D. The HunyuanWorld 1.0 authors give geometric consistency as the advantage of 3D-based methods over video-based ones.
Results move into existing tools. HunyuanWorld 1.0 exports meshes, and Marble exports Gaussian splats, meshes and videos.
A recorded drive can be replayed with changes. Waabi's UniSim converts a single recorded log into a closed-loop multi-sensor simulation in which actors can be added, removed or moved.
Generation can be fast enough for interactive editing. WonderWorld generates connected 3D scenes in less than 10 seconds on a single A6000 GPU.

Limits

There is less 3D training data than video. The HunyuanWorld 1.0 authors say 3D-based methods struggle with limited training data and memory-inefficient representations.
Fast movement is hard to represent. The 2025 embodied AI survey reports that scenes built from renderable pieces such as Gaussian splats have limited ability to handle rapid dynamics or changes in shape.
Physics support is basic in current products. World Labs describes Marble's collider meshes as low-fidelity meshes intended for coarse physics simulation.
Text and single-image prompts give limited control over details, as World Labs states for Marble.
Accurate 3D needs depth sensors or many camera views, which the 2026 robotic manipulation survey notes are less available than ordinary video.

Examples

UniSim (A Neural Closed-Loop Sensor Simulator)Waabi; University of Toronto; MIT · 2023
OccWorldTsinghua University · 2023
WonderWorldStanford University; MIT · 2024
HunyuanWorld 1.0Tencent Hunyuan · 2025
MarbleWorld Labs · 2025

Learned and hybrid physics simulators

A learned physics simulator is a neural network trained to predict the next physical state of a system, such as the positions and speeds of particles, mesh points or robot joints. It takes the current state and any forces or commands and outputs the state a short time later, and repeating this step produces a full motion. It is used to replace or speed up a traditional physics engine, and in robotics to train controllers inside the learned simulator and to adjust the simulator with real-world data.⁴⁹¹¹⁺¹ Inferred

How it works

Run a traditional physics simulator, or record real measurements, to collect many examples of a state and the state one time step later.
Describe the system as a graph in which particles, mesh points or robot parts are nodes and nearby or connected pieces are linked.
Train a network that passes messages along the links and predicts how each node moves in the next step.
Apply the network repeatedly to roll the simulation forward for hundreds or thousands of steps. GNS adds noise to its training data to limit the build-up of errors.
In robotics, plug the learned model into a simulator in place of the hand-written dynamics and contact code, train policies there and fine-tune the model with real robot data, as NeRD does.

Takes actions

Optional

Used for

speeding up physics simulationscience and engineering simulationrobot simulationtraining policies

Strengths

It can run faster than the solver it learned from. MeshGraphNets runs 1-2 orders of magnitude faster than the simulation on which it is trained.
It can handle larger systems than it was trained on. GNS generalises to thousands of timesteps and at least an order of magnitude more particles at test time.
It can learn from real measurements. The NeRD authors report that their learned simulators can be fine-tuned from real-world data, unlike most classical simulators.
The outputs are physical quantities such as positions and speeds, so they can be compared directly with measurements.

Limits

Errors build up over long simulations, so training needs extra measures such as the added noise used in GNS.
Many learned simulators need training for each application and do not transfer to new tasks or environments, as the NeRD authors state.
The examples listed here take the system as particles, meshes or joint states and do not start from raw camera images.
Training examples usually come from a classical simulator, so the learned model can be no more accurate than that data.

Examples

Interaction NetworksGoogle DeepMind · 2016
GNS (Learning to Simulate Complex Physics with Graph Networks)DeepMind; Stanford University · 2020
MeshGraphNetsDeepMind · 2020
NeRD (Neural Robot Dynamics)NVIDIA; University of Washington · 2025

How recent surveys group world models

Survey	Their groups	How they match the types on this page
A Comprehensive Survey on World Models for Embodied AI	Functionality axis: Decision-Coupled (task-specific) and General-Purpose (task-agnostic) world models, Temporal axis: Sequential Simulation and Inference (one step at a time) and Global Difference Prediction (many future steps at once), Spatial representation axis: Global Latent Vector; Token Feature Sequence; Spatial Latent Grid (bird's-eye-view or voxel grids); Decomposed Rendering Representation (NeRF, 3D Gaussian splatting)	The survey sorts each model along three separate axes instead of naming families, so each of our families matches a combination of cells. Our latent-dynamics family matches decision-coupled, step-by-step models with a global latent vector, where the survey lists World Models 2018, PlaNet and Dreamer. Our video-generation family matches general-purpose models that predict many steps at once with token features, where the survey lists Sora. Our action-video family matches general-purpose step-by-step models with token features, where the survey lists iVideoGPT, Genie and Vid2World, and it moves to the decision-coupled side when a video model is tied to one robot task. The survey lists V-JEPA and V-JEPA 2 in the same cell as Sora, so it does not separate feature prediction from pixel prediction at the top level. Our 3d-world family matches two of its representation classes, spatial latent grids (for example the occupancy model OccWorld) and decomposed rendering representations (for example GaussianWorld). The survey has no class for real-time interactive worlds and does not discuss GameNGen or DIAMOND. It also has no class for learned physics simulators, and physics-based models such as PIN-WM and ParticleFormer appear inside its decision-coupled cells.
Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond	Capability levels: L1 Predictor (one-step prediction); L2 Simulator (multi-step, action-conditioned rollouts that respect domain laws); L3 Evolver (revises its own model when predictions fail against new evidence), Governing-law regimes: physical, digital, social, scientific, Physical-world system types: physics simulation (classical engines); video generation models, split into appearance-first video generation, action-conditioned and interactive video worlds, and decision-oriented video world models; robotics and sim-to-real transfer; spatial reasoning; 3D-structured world models; autonomous driving world models; game world models (placed between the physical and digital regimes), Architecture axes: representation (symbolic, latent continuous, structured 3D, discrete tokens); dynamics (stochastic latent, deterministic value-aware, autoregressive token, diffusion-based); control interface	All seven of our families fall inside its physical regime, with games placed between the physical and digital regimes. Its L1 to L3 levels describe how capable a model is and cut across our families, so any of our families can contain L1 or L2 systems. Its group of action-conditioned and interactive video worlds covers both our action-video and interactive-world families, and lists Genie, GAIA-1, Oasis and Matrix-Game 3.0 together. Its appearance-first video generation group (Sora, Lumiere, VideoPoet) matches our video-generation family. Its architecture section separates stochastic latent dynamics (DreamerV3) from deterministic value-aware dynamics (MuZero, TD-MPC2), which are both in our latent-dynamics family, and it groups V-JEPA2 with DreamerV3 as latent continuous representations. Its 3D-structured world models (Marble, RTFM, TesserAct, RoboOccWorld) and its driving occupancy models (OccWorld) match our 3d-world family. Its physics simulation heading covers classical engines such as MuJoCo and Isaac Lab, which it says are not learned simulators, so our learned-simulator family has no direct match; the closest material is its fine-grained physical representations (ParticleFormer, GWM) and its scientific-regime operator learning. Its digital, social and scientific regimes (web and GUI agents, social simulation, AI for science) are outside our scope.
World Models for Robotic Manipulation: A Survey	Representation families: image and video; learned latent; motion fields and scene flow; geometric and spatiotemporal; physics-informed dynamics, Functional taxonomy: integrated prediction-action models; explicit predictive planners, Infrastructure roles: synthetic experience generation; candidate-action filtering and refinement; search-based action evaluation; learned environments for policy evaluation and improvement; outcome scoring and feasibility verification, Learning lifecycle: pretraining; post-training; inference adaptation	Its image and video family covers our video-generation and action-video families, and lists UniPi, SuSIE, GR-1, DreamGen, WorldGym and Genie Envisioner there. Its learned latent family covers our latent-dynamics family and also contains V-JEPA 2, so it does not separate our predictive-representation family. Its geometric and spatiotemporal family matches our 3d-world family, with a focus on predicting how a 3D scene changes under robot actions (TesserAct, PointWorld, GWM). Its physics-informed dynamics family overlaps our learned-simulator family and also includes hybrids that add learned parts to differentiable physics (PIN-WM). Its motion fields and scene flow family (FLIP, FlowVLA) has no counterpart in our list. Its integrated prediction-action models (GR-1, GR-2, WorldVLA) have no counterpart either, and we propose a world-action family for them. The survey covers robot manipulation only, so real-time interactive worlds for games are absent. Its infrastructure roles match our used_for values. Synthetic experience generation corresponds to generating training data, and learned environments correspond to evaluating and training policies.

Who builds world models

36 organisations are listed here. The table shows which types of world model each one has released. A filled dot means at least one of those models has public weights.

Organisation	Latent dynamics	Predictive representation	Action-conditioned video	World-action	Video generation	Interactive worlds	3D and 4D worlds	Learned physics	Largest disclosed round
Large technology companies
Alibaba (Tongyi Wan, DAMO Academy)China									Not disclosed
Ant Group (Robbyant / LingBot)China									Not disclosed
ByteDance (Seed)China									Not disclosed
Google DeepMindEurope									Not disclosed
Kuaishou (Kling)China									Not disclosed
Manycore Tech (SpatialVerse)China									Not disclosed
Meta (FAIR)North America									Not disclosed
Microsoft (Research and Xbox)North America									Not disclosed
NVIDIANorth America									Not disclosed
OpenAINorth America									Not disclosed
SenseTimeChina									Not disclosed
Skywork AI (Kunlun Tech)China									Not disclosed
Tencent (Hunyuan)China									Not disclosed
TeslaNorth America									Not disclosed
Start-ups
AMI Labs (Advanced Machine Intelligence)Europe									$1.03 billionMar 2026
DecartSeveral regions									$300 millionMay 2026
Dynamics Lab									—
General IntuitionNorth America									$320 millionJun 2026
GigaAI (极佳科技)China									—
Luma AINorth America									—
OdysseyNorth America									$310 millionJun 2026
RunwayNorth America									$315 millionFeb 2026
Shengshu Technology (生数科技, Vidu)China									—
WaabiNorth America									$750 millionJan 2026
WayveEurope									$1.2 billionFeb 2026
World LabsNorth America									$1 billionFeb 2026
Robot companies
1X TechnologiesSeveral regions									$100 millionJan 2024
AgiBot (智元机器人)China									—
Figure AINorth America									$1 billionSep 2025
Galbot (银河通用)China									—
Physical IntelligenceNorth America									$600 millionNov 2025
Skild AINorth America									$1.4 billionJan 2026
Unitree Robotics (宇树科技)China									—
WaymoNorth America									$16 billionFeb 2026
Research labs
Beijing Academy of Artificial Intelligence (BAAI)China									—
Shanghai AI LaboratoryChina									—

Disclosed funding

Each bar is the largest funding round an organisation has disclosed in US dollars. This is money raised by the whole company, not money spent on world models. Select a bar to see every round and its exact wording. The first bar is shortened, shown by the break, so that the others stay readable. A round marked Secondary was found only in news reports. Large technology companies do not disclose how much they spend on world models.

Funding rounds of the Chinese start-ups in the table (AgiBot, Galbot, Unitree, GigaAI, Shengshu Technology and Manycore Tech) have not been checked yet, so their funding is left blank.

Where world models are used

14 uses are listed. 2 are at the research stage, 6 are in pilots and 6 are offered as products.

Research2

Shown in papers or demos.

Pilot6

Used by a company in its own work, or tested with customers.

Product6

Offered to customers.

How world models are tested

Benchmarks for world models check how realistic the generated video is, whether it follows physics, or whether a policy gets the same result inside the world model as on a real robot.

Do world models rank robot policies the way real robots do?

Each row is one published comparison between scores inside a world model and scores on real robots. The value is the Pearson correlation r: 1.0 means the two sets of scores rise and fall together exactly. A hollow dot is the figure reported in the world model’s own paper. A filled dot is a figure measured by another team. A line shows a range of per-task values. Select a row to open its source.

Sources: the papers linked from each row^383639385+41. Values are copied from each paper. Studies compare different policies and tasks, so the values are not directly comparable.

5 more comparisons report no correlation

Ctrl-World vs real-robot rollouts on the authors' DROID setup. No correlation coefficient or MMRV reported. Linear fits of world-model rate on real rate across policy-task pairs (Figure 7): instruction following y = 0.87x - 0.04; success rate y = 0.81x - 0.11.^40240181

1X World Model outcome-prediction alignment. Alignment 63.06% when trained on about 216M Shelf video tokens; 71.17% with an added about 1.46B Arcade video tokens³⁹¹

GigaWorld-1 closed-loop success-rate alignment with real robots. No correlation statistic. Fitted line of generated vs real success rate: y = 1.134x - 0.091 (Fig. 16). Generated minus real success per subtask ranges from -0.13 to +0.07 across 12 subtasks (Fig. 17).^452645646+1

EnerVerse-AC vs real-robot success (four tasks, three training steps). No statistic; bar charts show the same task ranking and the same training-step trend. Real vs EVAC per task: Take a Bottle 28% vs 25%; Take a Toast 100% vs 90%; Take a Bacon 85% vs 88%; Take a Leaf 55% vs 50%. Per training step (Take a Bottle): 4K 40% vs 40%; 8K 61% vs 63%; 13K 79% vs 76% (Fig. 7).^373374372

RoboWM-Bench real-to-sim outcome consistency. Success consistency 10/10 and failure consistency 10/10 for every task (Pick Object, Pull Object, Push Object, Put on Plate, Discard Trash, Close Drawer, Put in Drawer): 140 of 140 outcomes matched⁴⁸²⁴⁸³

Benchmarks for world models 73

Benchmark	What it measures	Built by	Released
1X World Model Challenge Entry on this site	How well models predict future first-person frames of 1X's EVE humanoid from past frames and actions.³⁵²³⁵³	1X Technologies (2025 phases with OpenDriveLab)	Jun 2024
EVA-Bench (Embodied Video Anticipation Benchmark)	Embodied video anticipation by world models that combine a vision-language model and a video generator, across four meta-tasks: Action-Description, How-To, Finish-Thinking and Next-Step, on real robots, simulated robots and egocentric human activities, with in-domain and out-of-distribution samples.^354355356	Hong Kong University of Science and Technology, Peking University (State Key Laboratory of Multimedia Information Processing)	20 Oct 2024
WorldSimBench	Video generation models used as world simulators for embodied agents, in three scenarios: an open-ended embodied environment (Minecraft via MineRL), autonomous driving (CARLA) and robot manipulation (CALVIN). It checks perceived visual quality per embodied dimension and whether generated videos can be turned into correct control signals.^357358359+3	arXiv and project page: The Chinese University of Hong Kong, Shenzhen; Shanghai Artificial Intelligence Laboratory; Beihang University; The University of Hong Kong. PMLR version adds Sun Yat-sen University, University of Oxford and the Guangdong Key Laboratory of Big Data Analysis and Processing.	23 Oct 2024
WorldModelBench	Whether text-to-video and image-to-video generators behave as world models in seven application domains: autonomous driving, robotics, human activities, industrial, natural scenes, simulation gaming and animation. It checks instruction following, physics adherence and commonsense (frame-wise and temporal quality).^363364365+4	UC Berkeley; UC San Diego; NVIDIA; MIT	28 Feb 2025
EWMBench Entry on this site	How closely robot-manipulation videos from a video generator match real AgiBot World episodes in scene, motion and task meaning.³⁷⁰	AgiBot; Shanghai Jiao Tong University; MMLab-CUHK; Harbin Institute of Technology (arXiv header)	May 2025
AgiBot World Challenge, World Model track Entry on this site	How well models predict robot head-camera video from actions on held-out AgiBot World episodes.³⁷¹	AgiBot (2025 with OpenDriveLab)	May 2025
EnerVerse-AC (EVAC) as policy evaluator	Whether success rates of a Go-1 policy evaluated inside the EVAC action-conditioned world model match real-robot success rates across four retrieval tasks and across three training steps.^372373374+3	AgiBot; Shanghai Jiao Tong University; MMLab, CUHK	14 May 2025
DreamGen Bench	How well image-to-video world models adapt to a target robot embodiment and generalise to unseen objects, behaviours and environments: whether generated robot videos follow the instruction and obey physics. Setups: RoboCasa (simulated Franka) and three real Fourier GR1 humanoid splits (Object, Behavior, Environment).^54377378+4	NVIDIA (GEAR lab, lead), University of Washington, KAIST, UCLA, UCSD, Caltech, NTU, University of Maryland, UT Austin	19 May 2025
WorldEval	Whether success rates of real-robot manipulation policies, obtained by rolling each policy out inside an action-conditioned video world model, track and rank the same policies' real-robot success rates.^298383299+1	Midea Group; East China Normal University	25 May 2025
WorldGym	Whether success rates of VLA policies, obtained by Monte Carlo rollouts in an autoregressive action-conditioned video world model started from real first frames, match real-robot success rates and keep policy rankings. Also used to test policies on edited out-of-distribution scenes and instructions.^297385386+3	Stanford University; NYU; Google DeepMind	31 May 2025
Real-robot goal-image planning test (V-JEPA 2-AC)	Whether an action-conditioned world model can drive a real robot by planning: the robot is given goal images and the world model is used to search for actions, with no task-specific training or reward.³⁹⁰	FAIR at Meta (V-JEPA 2 paper)	Jun 2025
1X World Model evaluation	How often the world model's success/failure prediction matches real outcomes, and whether its scores pick the same checkpoints and architectures as real double-blind A/B evaluations on humanoid robots.³⁸³⁹¹	1X Technologies	16 Jun 2025
WM-ABench	Whether vision-language models (not video generators) behave as internal world models, using atomic tests of perception (visual, spatial, temporal, quantitative, motion) and prediction (mechanistic simulation, transitive inference, compositional inference), with controlled counterfactual simulations.^392393394+1	Maitrix.org, UC San Diego, Johns Hopkins University, Cornell Tech, EPFL, University of Michigan	27 Jun 2025
IRASim policy evaluation (LIBERO)	Whether success rates of diffusion-policy checkpoints judged in IRASim-generated rollouts match the success rates measured in the LIBERO MuJoCo simulator.^396397398+1	Hong Kong University of Science and Technology; ByteDance Seed	29 Jul 2025
PAI-Bench (Physical AI Bench) Entry on this site	Video generation and video understanding on physical-AI clips in six domains: common sense, driving, robots, industry, people, physics.⁴⁰⁰	Georgia Tech; Carnegie Mellon University (NVIDIA acknowledged for support)	Sep 2025
Ctrl-World	Whether the instruction-following rate and the task success rate of generalist DROID policies, rolled out closed-loop inside a multi-view action-conditioned video world model, match the same policies' rates on a real robot in a new DROID setup.^80401402+5	Stanford University; Tsinghua University	11 Oct 2025
Cosmos-Surg-dVRK	Whether success rates of surgical robot policies, rolled out online in a fine-tuned Cosmos world foundation model, match success rates of the same policies on a real da Vinci Research Kit (dVRK Si).^406407408	NVIDIA; Johns Hopkins University; Stanford University	17 Oct 2025
World-in-World	Whether visual world models help an embodied agent succeed in closed-loop tasks: Active Recognition (AR), Image-Goal Navigation (ImageNav), Active Embodied Question Answering (A-EQA) and robotic manipulation. Each world model is plugged into the same proposal-simulation-revision planning loop through a unified action API (text prompt, camera trajectory or low-level actions).^409410411+4	Johns Hopkins University (lead; corresponding author Jieneng Chen), Peking University, Princeton University, MIT, Harvard University	20 Oct 2025
Scalable Policy Evaluation with Video World Models (NVIDIA)	Whether success rates predicted by action-conditioned video models (post-trained Cosmos-Predict2-2B) with a VLM success judge match simulator success rates on four RoboMimic tasks and real success rates of three generalist policies on four Bridge-setup tasks.^416417418	NVIDIA Research; University of Toronto; Vector Institute	14 Nov 2025
Veo (Robotics) policy evaluation	Whether success rates of Gemini Robotics On-Device policy checkpoints, predicted by closed-loop rollouts in a Veo-based action-conditioned multi-view video model, match real ALOHA 2 success rates in nominal scenes and in edited out-of-distribution scenes, and whether the model finds unsafe behaviour that also occurs on the real robot.^87419420	Google DeepMind (Gemini Robotics Team)	11 Dec 2025
PolaRiS Entry on this site	Whether policy scores in simulated copies of real scenes, built from short video scans with 2D Gaussian splatting, match real-robot scores of generalist DROID policies.^421422423+2	University of Washington; Princeton University; UC Berkeley; Stanford University; Toyota Research Institute; University of Southern California; Cornell University; Physical Intelligence	18 Dec 2025
RBench Entry on this site	Whether video generators produce correct and physically plausible robot task videos across five task types and four robot body types.⁴²⁶	Peking University; ByteDance Seed	Jan 2026
WoW-World-Eval (Wow, wo, val)	Image-plus-text-to-video generation for robot manipulation, framed as an 'Embodied Turing Test' over five abilities: perception, planning, prediction, generalisation and execution. Includes a human Turing test (can people tell generated from real video) and an inverse-dynamics-model (IDM) Turing test (can actions recovered from generated video be executed on a real robot).^427428429	Peking University (State Key Laboratory of Multimedia Information Processing, School of Computer Science), Beijing Innovation Center of Humanoid Robotics, The Hong Kong University of Science and Technology	7 Jan 2026
DreamDojo policy evaluation (AgiBot fruit packing)	Whether the success rates of policy checkpoints simulated inside the DreamDojo world model match the success rates of the same checkpoints on a real robot, on one long-horizon fruit-packing task.^93430431+2	NVIDIA (lead); co-author affiliations also list HKUST, UC Berkeley, KAIST, University of Toronto, UC San Diego, University of Washington, Stanford, UT Austin	6 Feb 2026
WorldArena	How well embodied world models (text- or action-conditioned robot video models) predict bimanual manipulation video, and whether they are useful for three downstream jobs: generating training data for a policy (data engine), standing in for a simulator when scoring policies (policy evaluator), and planning actions (action planner). Human ratings are collected as a third view.^433434435+10	Tsinghua University (lead; corresponding author Yong Li), Shanghai Jiao Tong University, The University of Hong Kong, Princeton University, Chinese Academy of Sciences, University of Science and Technology of China, Peking University, National University of Singapore	9 Feb 2026
GWM-Robotics policy evaluation	Whether progress scores of VLA policies rolled out inside Runway's GWM-Robotics world model rank the policies in the same order as their real-world RoboArena evaluations.²⁹⁰²²⁴	Runway	27 Feb 2026
GigaBrain Challenge 2026 - World Model Track (CVPR 2026 workshop)	World models as evaluators of a VLA policy (GigaBrain) on 8 real-robot manipulation tasks: video quality when replaying teleoperation actions, and whether closed-loop rollouts driven by policy actions reach the same outcome as the real-robot reference video.^446447448+4	GigaAI (organizers Zheng Zhu, Xiaofeng Wang) with co-organizers from University of Hong Kong, Peking University, Shanghai Jiao Tong University, RoboChallenge/Dexmal and Horizon Robotics	Mar 2026
WorldArena Challenge @ CVPR 2026	Two tracks built on WorldArena: Track 1 video perception quality of embodied world models; Track 2 whether generated worlds work as data engines and as policy evaluators.^453454437+4	Organisers listed on the challenge page: Amap CV Lab (AMAP), Manifold AI, Tsinghua University, Princeton University, National University of Singapore, The University of Hong Kong	Mar 2026
PlayWorld policy evaluation	Whether success rates predicted by a video world model trained on autonomous robot play match real-robot success rates for many manipulation policies on contact-rich tasks, and whether predicted failure modes match observed ones.^456457458+4	Princeton University	9 Mar 2026
Interactive World Simulator sim-to-real policy evaluation	Whether task scores of imitation policies (final and intermediate checkpoints of DP, ACT, pi0 and pi0.5) evaluated closed-loop inside the learned world model match their real-robot scores on four ALOHA tasks.^463464465+4	Columbia University; Toyota Research Institute; Amazon; University of Illinois Urbana-Champaign	9 Mar 2026
PersistWorld world-model-to-real policy evaluation	Whether the task progress of robot policies rolled out inside an action-conditioned video world model matches their real-robot task progress, comparing PersistWorld (Ctrl-World post-trained with reinforcement learning on its own rollouts) with the base Ctrl-World.^470471472+3	Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague	26 Mar 2026
Cosmos-H-Surgical-Simulator evaluation on Open-H-Embodiment	How closely an action-conditioned surgical world model's generated video follows the recorded kinematic actions of held-out real surgical-robot episodes (open-loop replay), as groundwork for in-silico policy evaluation.^476477478+2	NVIDIA with the Open-H-Embodiment Consortium (50+ institutions in the paper's author list); model card by NVIDIA	Apr 2026
RoboWM-Bench	Whether manipulation behaviour in videos generated by world models can be executed: generated human-hand or robot-arm videos are converted to robot actions and run in simulated scenes, including real-to-sim reconstructions of real tabletop scenes.^481482483+2	Peking University (lead; corresponding author Ruihai Wu), Tsinghua University, Lightwheel	21 Apr 2026
dWorldEval policy evaluation proxy	Whether success rates estimated inside a discrete-diffusion world model (with a progress token that marks task completion) match ground-truth success for pi0 checkpoints on LIBERO, heterogeneous policies on RoboTwin, and three policies on five real bimanual AgileX tasks; it also compares three video-diffusion world models on LIBERO.^486487488+2	Current Robotics; University of Toronto	24 Apr 2026
Cosmos-HumanEval (Cosmos HUE) Entry on this site	Human judgement of physical laws, visual integrity, semantic alignment and geometry in generated videos.¹³⁸⁴⁹¹	NVIDIA	May 2026
WorldArena 2.0	Extends WorldArena along three axes: visuotactile prediction (tactile plus video) for contact-rich tasks, world models used as interactive reinforcement-learning environments for policy improvement, and evaluation across two simulators and a real robot (RoboTwin 2.0, LIBERO, AgileX split-type ALOHA).^492493494+4	Tsinghua University (lead; corresponding author Yong Li), Shanghai Jiao Tong University, Zhejiang University, Stanford University, The University of Hong Kong, Princeton University, Chinese Academy of Sciences, University of Science and Technology of China, Peking University, National University of Singapore	18 May 2026
OSCAR policy evaluation on RoboArena	Whether OSCAR-generated replays of RoboArena episodes, judged by GPT-5, reproduce the real per-policy success rates and ranking of seven open-source DROID policies.⁴⁹⁹⁵⁰⁰	Peking University; University of Michigan; NVIDIA	3 Jun 2026
WEAVER policy evaluation on real hardware	Whether success judged on rollouts imagined by the world model matches the real success rates of two pi0.5-based policies on five real manipulation tasks, compared with Ctrl-World and with the WEAVER model before task fine-tuning.^501502503+5	Mila - Quebec AI Institute; Universite de Montreal; Carnegie Mellon University; McGill University	11 Jun 2026
RoboWorld Entry on this site	Whether scores from closed-loop rollouts of DROID policies in an autoregressive video world model match the RoboArena real-world leaderboard.^509510511	KAIST; Config	1 Jul 2026
WMBench (GigaWorld-1)	How well video world models serve as surrogate evaluators of robot policies, by comparing generated rollouts with matched real-robot executions on eight manipulation tasks built from teleoperation data and GigaBrain policy rollouts.^295452512+3	GigaAI; Tsinghua University	2 Jul 2026
WorldArena 2.0 Challenge @ IROS 2026	Three tracks: Track 1 video quality of embodied world models with harder tasks and new out-of-distribution scenes; Track 2 world models as RL environments for policy optimisation; Track 3 real-world manipulation by world action models (WAM), in tactile and vision-only settings.^514498497+2	WorldArena 2.0 team (organisers not named on the challenge page; contact worldarenav2@outlook.com); paper team led by Tsinghua University	10 Jul 2026
TriWorldBench	Whether an embodied world model's synchronized head, left-wrist and right-wrist videos describe one consistent manipulation event, plus task alignment, physical and 3D coherence, motion quality, temporal consistency and visual quality.^515516517+4	Peking University, Tsinghua University, Beihang University, Shanghai Jiao Tong University, University of Science and Technology of China, Shanghai AI Laboratory (arXiv affiliation block); the website lists OpenCompass in place of Shanghai AI Laboratory	28 Jul 2026
Pelican-Sim 1.0 policy evaluation and ranking (RoboTwin)	Whether the Pelican-Sim world model combined with a fine-tuned VLM judge reproduces RoboTwin simulator success rates and the ranking of five checkpoints from one VLA training run.^522523524+1	Beijing Innovation Center of Humanoid Robotics (X-Humanoid), WFM System Group	10 Sep 2026
DexTouch-WM world models as policy evaluators	Whether scores of three VLA policies rolled out closed-loop inside two task-adapted visuo-tactile world models match their scores on a real dexterous-hand robot on four tasks.⁵²⁶⁵²⁷	HKUST (Guangzhou); Xspark AI; Peking University; Tsinghua University; University of Hong Kong	17 Sep 2026
VBench (including VBench++)	General video generation quality split into 16 dimensions. Video Quality: subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality, imaging quality. Video-Condition Consistency: object class, multiple objects, human action, color, spatial relationship, scene, appearance style, temporal style, overall consistency. VBench++ adds image-to-video evaluation with an Image Suite, long-video evaluation, and trustworthiness (culture fairness, human bias, safety). There is no physics dimension.^528529530+4	S-Lab, Nanyang Technological University; Shanghai Artificial Intelligence Laboratory; Nanjing University; The Chinese University of Hong Kong	29 Nov 2023
VideoPhy	Whether text-to-video outputs follow the caption (semantic adherence) and physical commonsense in everyday interactions between materials: solid-solid, solid-fluid and fluid-fluid, with captions marked easy or hard by graphics researchers.^535536537+2	University of California Los Angeles; Google Research	5 Jun 2024
PhyGenBench (with PhyGenEval)	Whether text-to-video outputs show the correct physical phenomenon for prompts that each target one physical law, in four domains: mechanics, optics, thermal and material properties. Semantic alignment with the prompt is scored separately.^540541542+1	Shanghai Jiao Tong University; OpenGVLab, Shanghai AI Laboratory; The University of Hong Kong; The Chinese University of Hong Kong	7 Oct 2024
Physics-IQ	Whether a video generator can predict how a real, filmed physical experiment continues. Scenarios cover solid mechanics, fluid dynamics, optics, thermodynamics and magnetism. Each 8-second real video is split into a 3-second conditioning part and a 5-second test part. Video-to-video models get the 3 seconds; image-to-video models get only the last conditioning frame (the 'switch frame'); both can also get a text description that does not reveal the outcome.^544545546+2	Google DeepMind; INSAIT, Sofia University (first author, work done while at Google DeepMind)	14 Jan 2025
VideoPhy-2	Physical commonsense and prompt adherence of text-to-video outputs for real-world actions (sports and physical activities, object interactions), plus whether specific physical rules for each video are followed.^549550551+3	University of California Los Angeles; Google Research	9 Mar 2025
IPV-Bench (Impossible Videos)	Whether video generators can follow prompts for impossible events that defy physical, biological, geographical or social laws while keeping high visual quality; separately, whether Video-LLMs understand impossible videos.^555556557+1	Show Lab, National University of Singapore	18 Mar 2025
VBench-2.0	'Intrinsic faithfulness' of generated video in five dimensions with 18 capabilities: Human Fidelity (human anatomy, identity, clothes); Creativity (diversity, composition); Controllability (dynamic spatial relationship, dynamic attribute, motion order understanding, human interaction, complex landscape, complex plot, camera motion); Physics (State Change: mechanics, thermotics, material; Geometry: multi-view consistency); Commonsense (motion rationality, instance preservation).^559560561+1	Shanghai Artificial Intelligence Laboratory; S-Lab, Nanyang Technological University; Sun Yat-sen University; The Chinese University of Hong Kong	27 Mar 2025
WorldScore	World generation from an image plus a camera-trajectory layout, posed as a sequence of next-scene tasks, so that 3D scene generators, 4D generators and video generators can be compared. Three aspects: controllability, quality and dynamics, over static worlds (indoor and outdoor) and dynamic worlds (5 motion types).^562563564+3	Stanford University	1 Apr 2025
Morpheus	Whether video generators conditioned on frames of real, filmed Newtonian rigid-body experiments continue them in line with the governing equations of motion and with conservation of energy, momentum and period.^568569570+1	University of Amsterdam; University of Trento	3 Apr 2025
T2VPhysBench	Whether text-to-video systems obey 12 physical laws in three groups: Newton's three laws and gravitation; conservation of energy, mass, linear and angular momentum; and phenomenological laws (Hooke's law, Snell's law, law of reflection, Bernoulli's principle). Also tests prompts with added law-specific hints and counterfactual prompts that ask for physics-breaking videos.⁵⁷²⁵⁷³	Guilin University of Electronic Technology; University of Arizona; University of Wisconsin-Madison; Simons Institute for the Theory of Computing, UC Berkeley; Arizona State University	1 May 2025
IntPhys 2 Entry on this site	Whether video models can tell physically possible from impossible events in synthetic videos.⁵⁷⁴	FAIR at Meta	Jun 2025
MVPBench (Minimal Video Pairs) Entry on this site	Physical understanding of video-language models, with near-identical video pairs that have opposite answers.⁵⁷⁵	FAIR at Meta	Jun 2025
CausalVQA Entry on this site	Cause-and-effect reasoning of video-language models on real Ego-Exo4D clips: counterfactual, anticipation and planning questions.⁵⁷⁶	FAIR at Meta	Jun 2025
WorldPrediction	High-level world modeling and long-horizon procedural planning on human activity video. WorldPrediction-WM: given an initial and a final state image, pick the action clip that causes the change. WorldPrediction-PP: pick the correctly ordered sequence of 3-10 action clips. Candidate actions are shown in other scenes ('action equivalents') so background continuity cannot be used.^577578579	Meta FAIR Paris; The Hong Kong University of Science and Technology; ISIR Sorbonne Université	4 Jun 2025
PhyWorldBench	Physical realism of text-to-video output across 10 physics categories, from object motion and energy conservation to rigid-body interactions and human or animal motion, plus an Anti-Physics category whose prompts ask for physics-violating events.^580581582+1	University of California, Santa Cruz; NVIDIA Research; Northeastern University; University of California, Santa Barbara	17 Jul 2025
WorldMark	Interactive image-to-video world models driven by navigation commands: how motion reacts to each command (direction, purity, latency, stability, per translation and rotation axis), whether the generated world stays the same over time (world memory at three time scales), and visual quality, across first- and third-person views, real and stylized scenes, and three difficulty tiers.^584585586+1	Alaya Lab; The University of Tokyo; Shanghai Innovation Institute	23 Apr 2026
WBench	Interactive video world models in multi-turn use: video quality, adherence to a stated world setting, adherence to interactions (navigation, subject action, event editing, perspective switching), consistency, and physics compliance.^588589590+1	Fudan University; Meituan LongCat Team	25 May 2026
Human World Bench (HWB)	Egocentric image-to-video generation of human manipulation tasks under task-level instructions: whether the video shows the requested actions and objects, and whether motion, object dynamics, contact and hand anatomy are physically plausible.^99138592	NVIDIA (Cosmos 3 technical report)	1 Jun 2026
Physics-IQ Verified	The same task and the same 66 real-world scenarios as Physics-IQ, after an audit that corrected prompts and ground-truth motion maps, so that the score reflects physical prediction rather than prompt ambiguity or unrelated motion in the recordings.^593594595+3	Anates Labs; Technical University of Munich; University of Technology Nuremberg; Tuebingen AI Center, University of Tuebingen; Helmholtz AI, Munich; Google DeepMind (Jaini and Geirhos, described as advisory in the acknowledgements)	17 Jun 2026
PlayWorld	Long-horizon capability of interactive video world models when a player pursues a stated objective: geometry consistency, interaction fidelity, out-of-sight evolution (what happens to things while unseen) and insight evolution (processes that should change while watched), plus basic video quality and action controllability.^599600601+2	The Chinese University of Hong Kong; The University of Hong Kong; Zhejiang University; Kling Team, Kuaishou Technology (as listed in the arXiv HTML; the code repository now resolves to github.com/hku-sail/PlayWorld)	13 Aug 2026
DeepMind Control Suite	Continuous-control performance of reinforcement-learning agents on simulated MuJoCo bodies (cart-pole, cheetah, walker, humanoid, manipulator and others); model-based agents such as Dreamer report results on it.^604605606	DeepMind (paper title and repository owner google-deepmind)	Jan 2018
Atari 100k	How well a reinforcement-learning agent plays Atari games when it may interact with each game only briefly; world-model agents (SimPLe, Dreamer and others) use it to show that learning inside a model saves real interaction.^607606608	Introduced with SimPLe by Google Brain, deepsense.ai, Institute of Mathematics of the Polish Academy of Sciences, University of Warsaw, UIUC and Stanford (paper affiliations)	Mar 2019
Crafter	Breadth of abilities of an agent in a 2D open-world survival game: exploration, resource collection, crafting, survival.^609610606	Danijar Hafner (Google Research, Brain Team; University of Toronto)	Sep 2021
DriveArena	Open-loop and closed-loop driving performance of camera-based driving agents inside a generative simulation platform. A Traffic Manager simulates traffic on nuScenes, CARLA or OpenStreetMap road networks, and World Dreamer (a layout-conditioned diffusion model trained on nuScenes) generates the six surround-view camera images the agent sees at each step.^611612613+2	Shanghai Artificial Intelligence Laboratory; Zhejiang University; Shanghai Jiao Tong University; East China Normal University; Technical University of Munich	1 Aug 2024
ACT-Bench	Action fidelity of driving world models: whether a model that generates front-camera driving video from context frames plus a commanded trajectory actually shows the commanded motion.^616617618+2	Turing Inc.	6 Dec 2024
Bench2Drive-R	Reactive closed-loop evaluation of camera-based end-to-end driving models using real-world data: a rule-based behaviour controller built on the nuPlan simulator moves the other road users, and a diffusion-based generative renderer produces the camera images for each new state. Sensor rendering and behaviour control are separate modules. The paper also measures the renderer itself (image quality, layout adherence, temporal consistency).^621622623	Shanghai Jiao Tong University (Dept. of CSE, School of AI, and MoE Key Lab of AI)	11 Dec 2024
WorldLens	How well generative driving world models (multi-camera video generators conditioned on nuScenes scene layouts) produce video that looks real, keeps consistent 3D geometry, follows physics, lets a pretrained driving planner drive, and supports perception models trained on real data. Five aspects (Generation, Reconstruction, Action-Following, Downstream Task, Human Preference) with 24 dimensions in total.^624625626+4	WorldBench Team (the only group name on the arXiv paper, PDF title page and project page; no institutional affiliation block is given)	11 Dec 2025
DrivingGen	Generative video world models for driving (image-to-video, optionally conditioned on an ego trajectory): how real the video looks, how plausible the ego motion implied by the video is, temporal and per-agent consistency, and how closely the video follows a commanded ego trajectory, across varied weather, time of day, world regions and maneuvers.^631632633+2	University of Toronto; CUHK MMLab	4 Jan 2026
GameWorld Score	Minecraft world models that generate video from an initial image plus keyboard and mouse actions: visual quality, temporal quality, action controllability and physical rule understanding, in 8 dimensions.^636637638	Skywork AI	23 Jun 2025

Known problems

Almost every published agreement number between a world-model evaluator and real robots was measured by the world model's own builders, on 1 to 18 policies or checkpoints, often 3 to 8. Third-party checks without shared authors exist only for Ctrl-World (PersistWorld and WEAVER papers on real robots; dWorldEval and WorldArena against simulators), WorldGym (dWorldEval, simulator only), Cosmos-Predict 2.5 (WorldArena, simulator only) and IRASim (NVIDIA, real robots). Several studies reuse real-robot numbers from earlier work instead of running new paired trials (WorldGym from OpenVLA, RoboWorld and Runway from RoboArena).^385510290+1 Inferred
The same world model gets very different agreement scores depending on who tests it and where. Ctrl-World's agreement with ground truth was Pearson r = 0.53 / MMRV 0.22 in the PolaRiS real-robot study, r = 0.552 / MMRV 0.215 in the WEAVER real-robot study, r = 0.796 / MMRV 0.053 in the PersistWorld real-robot study, r = 0.841 against the LIBERO simulator in the dWorldEval study, and r = 0.986 against the RoboTwin simulator in WorldArena. Its own paper reports no coefficient.^422501470+3 Inferred
The evaluators that third parties cannot run are the ones built by companies. Google DeepMind's Veo (Robotics) evaluator, Runway's GWM-Robotics, 1X's world model and Wayve's GAIA-3 have no released code or weights for evaluation, so their agreement numbers cannot be reproduced; RoboWorld's code was marked 'coming soon' on its project page.^420290391+1 Inferred
World-in-World finds that visual quality does not predict closed-loop task success: on its leaderboard of 16 Active Recognition entries, zero-shot Cosmos-Predict2 has the highest generation-quality score (0.481732) but the lowest success rate (55.35%, tied with Wan2.2 5B), while controllability (1 - LPIPS) tracks success more closely.^410647412
Vision-language judges, which several world-model evaluators use to decide whether a task succeeded, are error-prone. On FailBench (2,197 manipulation attempts from 14 public sources, 12 real and 2 simulated), the best of 13 VLM-based detectors reached 0.77 mean balanced accuracy, fell below 0.60 on contact-intensive assembly, and leaned toward predicting success when evidence was ambiguous.⁶⁴⁸
World models can predict higher success than real robots achieve. DreamDojo's authors report that its absolute success rates are often higher than real ones; in its Fig. 5a, DreamDojo rates span about 0.06-0.81 while real rates span 0.0-0.44.⁴³⁰⁴³¹
Generated videos can outscore real videos on automated benchmarks. On the PAI-Bench generation leaderboard, Cosmos3-Super (83.9) and Cosmos3-Nano (83.7) score above the real source videos (82.6) overall, and in the robot domain Cosmos3-Super (90.0), Cosmos3-Nano (90.2) and Veo-3 (86.9) score above the source videos (86.2).⁶⁴⁹
Contact-rich, granular and deformable interactions remain hard to simulate. WEAVER names bean pouring and bag and towel handling as the hardest tasks; Veo (Robotics) reports objects appearing spontaneously during contact and its weakest out-of-distribution correlation on novel objects (Pearson 0.56).^502419650

86 problems are recorded in total. All of them are in the downloadable data.

Media to follow

Blogs, newsletters, channels and accounts to follow on world models. Each one posted something in the 90 days before 10 Oct 2026. These are our picks.

PodcastEN

Dream Machines: Physical AI, Robotics, Being Human

Evan Helda (Head of Physical AI at Nebius)

Interview podcast and essays on physical AI, simulation and world models.

Start withWorld Models: From Pixels to Physics w/ SpAItial CEO Matthias Niessner

dreammachines.ai9 Oct 2026 · Irregular

NewsletterEN

Ani's Substack

Anirudha Majumdar (Princeton University)

Newsletter of a Princeton robotics researcher; in 2026 it ran a five-part series on video models and two posts on how world models for robots should be built.

Start withUnderstanding Video Models: Part V - Evaluation

animajumdar.substack.com18 Aug 2026 · Irregular

YouTubeEN

Saturday Robotics: Robotics & World Model Reading Club

Saturday Robotics (San Francisco group)

Recorded reading-club sessions where researchers and builders present papers on robot world models, world action models and robot learning.

Start withRobotics & World Models Reading Club 18.1: Is Latent All You Need for World Action Models?

youtube.com5 Oct 2026 · Irregular

BlogEN

Google DeepMind Blog

Google DeepMind

Main blog of Google DeepMind, where Genie (interactive world models), Veo (video generation) and Gemini Robotics are announced alongside many unrelated topics.

Start withSimulate real-world places with Project Genie and Street View

deepmind.google6 Oct 2026 · About weekly

Bilibili中文

可lip (Bilibili)

可lip (AI postdoc at the National University of Singapore)

Chinese explainer videos on world models, world action models, video generation and VLA models, including a numbered 世界模型 series.

Start with世界模型第1期 · 梦的开始：从 World Models 到 DayDreamer，让策略在梦里练，机器狗真机 1 小时学会走路

space.bilibili.com9 Oct 2026 · About weekly

Website中文

量子位 QbitAI: 世界模型 tag

量子位 (QbitAI)

Chinese AI news site's topic page for world models, mirroring articles from the 量子位 WeChat account on new Chinese and international world models and benchmarks.

Start with五大高校联手发榜！首份机器人三视角世界模型评测结果出炉，榜单持续更新中

qbitai.com23 Sep 2026 · About weekly

YouTubeEN

World Labs (YouTube)

World Labs

Company channel with model announcements (Atlas), demos of generated 3D worlds, and tutorials for Marble worlds and Gaussian splats.

Start withIntroducing Atlas; A Foundation Model for Spatial Intelligence

youtube.com17 Sep 2026 · Irregular

BlogEN

NVIDIA Technical Blog: Cosmos tag

NVIDIA

Developer posts tagged Cosmos: how to post-train and use NVIDIA's Cosmos world foundation models for robots, autonomous vehicles and synthetic data.

Start withBeyond VLAs: How World Action Models Reshape Robot Manipulation

developer.nvidia.com29 Sep 2026 · Irregular

Bilibili中文

被迫成为理工男 (Bilibili)

被迫成为理工男

Numbered daily paper-reading videos on JEPA, world models, world action models and video generation.

Start withDay 22: Fast-WAM: Do World Action Models Need Test-time Future Imagination?

space.bilibili.com10 Oct 2026 · About weekly

BlogEN

World Labs: Research & Insights

World Labs

Company blog on 3D world models (Marble, Atlas, RTFM), real-to-sim-to-real worlds for robot training, and company news.

Start withAtlas: A World Model for Spatial Intelligence

worldlabs.ai28 Sep 2026 · Irregular

BlogEN

Odyssey: The latest from Odyssey

Odyssey

Company blog where Odyssey announces its world models (Odyssey-3, Agora-2, Starchild-1), its PROWL reinforcement-learning work and its CaliBench evaluation.

Start withIntroducing CaliBench: Are World Models Physically Calibrated?

odyssey.systems8 Oct 2026 · Irregular

BlogEN

Runway Research

Runway

Research hub for Runway's general world models (GWM), real-time interactive worlds, an interface world model (Solaris) and a world action model for robots (Praxis-1).

Start withIntroducing Praxis-1

runwayml.com30 Sep 2026 · Irregular