ALFRED

The paper and the project site present ALFRED as a benchmark with fixed splits, fixed metrics and a test server.

Authors: Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox.

University of Washington · Paul G. Allen School. Seven of the eight authors list it.²

Carnegie Mellon University · Language Technologies Institute (Yonatan Bisk).²

Allen Institute for AI · Yonatan Bisk, Winson Han, Roozbeh Mottaghi. The data bucket on Amazon S3 is named ai2-vision-alfred.²²⁰

NVIDIA · Second affiliation of Dieter Fox.²

arXiv v1 on 2019-12-03. Published at CVPR 2020.

The GitHub repository was created on 2019-12-12 (GitHub API).

Dataset release 2.1.0 (json, json_feat and full packages) for the AI2-THOR 2.1.0 simulator. No later dataset version exists.

AI2-THOR 2.1.0 · Pinned in requirements.txt together with PyTorch 1.1.0. Uploaded to PyPI on 2019-09-06. The latest AI2-THOR on PyPI is 5.0.0 (2022-12-13).¹⁶¹⁷

Paper v2 (2020-03-31) · Results table updated after the 2020-03-28 change that selects the target object by mask overlap (IoU).¹⁴

Errata for the Goto sub-goal · The Goto sub-goal success rule changed to 'within 3 planner steps'. The main results and the leaderboard are not affected.²²⁴

Quickstart data fix (2020-10-26) · A missing stop frame was added to the json_feat package.⁴²⁰

Newest leaderboard entry on 2026-09-28. Last repository commit on 2026-02-05 (README only). Since April 2025, teams email their results file and the authors post the scores.

The leaderboard still receives and posts entries in 2026. The code has not changed since 2021-08 apart from README edits and one dependency pin.

Commit file lists: every commit after 2021-08-29 touches only README.md or requirements.txt (a flask version pin authored 2023-02-28, merged 2025-04-23). The Ai2-hosted leaderboard was deprecated in April 2025 and replaced by email submissions. 23 issues are open (GitHub API).

AI2-THOR 2.1.0, a Unity-based household simulator from the Allen Institute for AI. The paper used AI2-THOR 2.0; the code requires 2.1.0.

An egocentric agent with 5 navigation actions (move ahead, rotate left or right, look up or down), 7 interaction actions (pick up, put, open, close, toggle on, toggle off, slice) that target objects through a pixel mask, and a stop action. It has no robot body model. 'virtual-agent' is the closest taxonomy value; 'mobile-manipulator' would overstate the physics. At test time, leaderboard entries may use only RGB images and language.

Room counts from the paper, Section 3. Bathrooms, bedrooms and living rooms are mapped to 'home'.

7 task types (for example Pick & Place, Heat & Place, Clean & Place, Examine in Light) over 2,685 combinations of object, receptacle and room

120 rooms in AI2-THOR. Splits by room: train 108, validation seen 88, validation unseen 4, test seen 107, test unseen 8.

Paper Table 2. Seen rooms are subsets of the training rooms.

8,055 expert demonstrations (3 per task setting) generated by a PDDL planner, with 25,743 crowd-written English directives. Average 50 steps; 428,322 image-action pairs.

Directives were written on Amazon Mechanical Turk by at least three annotators per demonstration and checked by a second group of workers.

Directive splits · Train 21,023; validation seen 820; validation unseen 821; test seen 1,533; test unseen 1,529 (paper Table 2)²

Planner-made, filtered · The organisers' retrospective says task configurations the PDDL planner could not solve were dropped, which leaves fewer corner cases than in TEACh.²⁴

Download sizes · Trajectory JSONs 35,748,602 bytes; JSONs with ResNet features 17,835,841,659 bytes; full package 116,073,781,310 bytes (HTTP Content-Length, read 2026-10-10)²⁰²¹

The headline 'unseen' test split uses 8 rooms that do not appear in training, with new object variants and newly written instructions. The 'seen' split reuses training rooms.

Paper Table 2 and Section 3: unseen validation (4 scenes) and unseen test (8 scenes) are distinct from training and from each other; the split examines generalisation 'to entirely new spaces with novel object class variations'. The 7 task types and 84 object classes are the same in training and test.

Task success is the headline. Goal-condition success gives partial credit. Both also come in a path-length-weighted form.

Task success: an episode counts only if every goal condition holds at the end, for example a potato slice is heated and lies on a counter. Goal-condition success: the share of goal conditions met (2.55 per task on average), which gives partial credit. Path-length-weighted (PLW) versions multiply each score by the expert's step count divided by the larger of the expert's and the agent's step counts, so taking twice as long as the expert halves the credit. The leaderboard ranks entries by success rate on the unseen test split.

Paper Section 5.1 and the leaderboard 'Scoring' section. Episodes end after 1,000 steps or 10 failed actions. Humans scored 91.0% task success and 94.5% goal-condition success on 100 unseen test directives (paper Section 6.4).

Teams run their agent locally on the test instructions and submit the action sequences; the server replays them in the simulator. The README forbids changing the step and failure limits.

The agent itself runs on the team's machine, so the organisers check input rules through a questionnaire (RGB and language only at test time) rather than by running the policy.

The leaderboard shows single numbers. One study reports a seed-to-seed spread of 3 points in success rate for the same model on unseen validation (s20).

Official leaderboard on askforalfred.com: 95 entries from 2020-03-28 to 2026-09-28. One submission per 7 days; all submissions are public; email submission since April 2025.

Row count and dates from the CSV embedded in the leaderboard page, read 2026-10-10.

LICENSE file: MIT, copyright 2019 ALFRED.

MIT, by our reading. The README's licence section says 'MIT License'; there is no separate data licence.

The repository does not say explicitly that MIT covers the data hosted on Amazon S3. We read the repository licence as covering it. The data README and download script name no other terms.

We found no separate licence for the scenes and object models in AI2-THOR's builds. We read the Apache-2.0 licence of the AI2-THOR repository as covering them. Not checked: third-party asset terms inside the Unity builds.

Direct downloads from Amazon S3, no registration. The three archive links answered HTTP 200 on 2026-10-10 (headers only). Test-set scores require emailing the results file.

Code MIT, data MIT by our reading, simulator Apache-2.0. All three allow commercial use with attribution. Asset coverage is inferred (see license_assets). Not legal advice.