SIMA 2 evaluation

Classified by the Atlas from how the authors describe and distribute it.

Title block: 'SIMA Team, Google DeepMind'. arXiv lists Adrian Bolton and 64 other authors.

2025-11 (announcement blog, 2025-11-13); paper arXiv v1 2025-12-04⁴

Evaluation lineage: SIMA 1 paper, arXiv 2404.10179 v1 2024-03-13⁵

Section 3.4 names 'SIMA Evaluation Suite 2.0'. Changes from SIMA 1: more domains, more tasks ('often by an order of magnitude' for programmatic evals), success text must persist several seconds, limits on actions after completion, sequential chains must be fully completed.

arXiv lists only v1. The PDF linked from the blog carries the date 2025-12-05.

Closed-loop keyboard-and-mouse control inside game engines and research environments. These are game engines, not physics simulators built for robotics (see taxonomy_friction).

Training: Construction Lab, Playhouse, WorldLab (research environments); Goat Simulator 3, Hydroneer, No Man's Sky, Satisfactory, Space Engineers, Valheim, Wobbly Life (commercial). Held-out: ASKA, MineDojo (Minecraft); The Gunk and Genie 3 worlds only qualitatively.

Blog lists more partner games (also Eco, The Gunk, SteamWorld Build, Road 96, Teardown) than the paper's 7 commercial training games; the paper's list is used for the evaluation.

Organiser-run, self-reported by Google DeepMind. Human baselines are shown with and without the agent's time limit. Uncertainty statistics are not described in the text. Values are the bar labels printed in Fig. 6 of the PDF. Bar-to-series mapping read from bar order; consistent with the text's 'SIMA 2 effectively doubles the average success rate of SIMA 1'. Baseline Gemini models without SIMA training: Flash-Lite 3.2%, Pro 7.0% over the 10 training domains (text, https://arxiv.org/html/2512.04797). Human figures are for 'a representative subset of tasks'.

Results appear only in the paper and blog. No public leaderboard found.

No evaluation code, task list or repository mentioned in the paper or blog. arXiv's CC BY 4.0 licence covers the paper text only.

No evaluation tasks or data released. Commercial games are third-party products.

SIMA 2 was released as a 'limited research preview' to a small cohort of academics and game developers. Games used under licensed agreements with their developers.