Habitat Navigation Challenge

Habitat Navigation Challenge (PointNav, ObjectNav, ImageNav)

How to read this picture

The Habitat Navigation Challenge was a yearly contest from 2019 to 2023. A simulated robot had to reach a given point or object inside scanned buildings it had not seen before.123

Sources
Last checked 10 Oct 2026Full entry53 of 64 facts checked at the sourceNext check 8 Apr 2027
Runs in
Simulation3
Checked against real robots
Checked
Measured by the organisers. The ObjectNav ranking was reversed in a real home.45
Skill
Navigation
Robot
Wheels only23
Used by
1,022 submissions from 54 teams in 20226
2,185 citations
Licence
MIT789
Commercial use: not allowed

What a score here does not tell you Inferred

  1. How often a policy (the robot's control model) will succeed on a real robot.In one test in a real home, the ranking from simulation was reversed.
  2. Whether models trained on it can be used commercially.Matterport's terms allow academic use only.
  3. How much methods improved from one edition to the next.The scenes, sensors and rules changed from year to year.

Comparisons with real robots

StudyResultWhat was comparedDone by
Sim2Real Predictivity (Kadian et al.)
Dec 2019		Under the 2019 rules, the Pearson correlation (SRCC) was 0.18 for success and 0.603 for SPL. After the simulator was tuned, it was 0.844 for success and 0.875 for SPL.4The same 9 PointNav models were run in a scanned copy of one lab and on a real LoCoBot robot in that lab. There were 405 runs on each side. The study’s authors described the result as “low”.The benchmark’s authors
Navigating to Objects in the Real World (Gervet et al.)
Dec 2022		The simulation order of 4 end-to-end variants was reversed. They scored 77% to 48% in simulation and 0% to 30% in the real home. The per-episode SRCC was 0.20 to 0.70.56 ObjectNav policies were scored on the 2022 challenge validation split and in one real home, with 10 episodes. 3 of them were also scored in 6 real homes, with 60 episodes. The study’s authors described the result as “often inversely proportional”.The benchmark’s authors

Benchmarks built on Habitat Navigation Challenge: HM3D-OVON, MultiON, HomeRobot OVMM Challenge.101112

Our assessment Opinion

ObjectNav scores are weak evidence of real-robot skill.

Reasoning

A high Habitat ObjectNav score says little on its own about success on a real robot. In the one real-home study, run by the organisers' own team, the simulation ranking of end-to-end policies was reversed, while modular methods did better on the robot than in simulation.

Confidence: medium

Check the dataset version, split and sensors before you compare numbers.

Reasoning

Compare Habitat navigation numbers only when the task version, scene dataset, split and sensors match. The rules and datasets changed in almost every edition, and since the servers closed, new numbers are self-reported validation runs.

Confidence: high

PointNav results no longer show differences between methods.

Reasoning

PointNav under the published Habitat rules is solved. A new PointNav result adds little information about a method.

Confidence: high

The 2020 rules drew on a real-robot study. The rules as a whole were never checked on a real robot.

Reasoning

The 2020 rule changes followed a study that measured how well simulation results matched real-robot results, which is unusual. That study tested agents that had GPS+compass and found that zero motion noise predicted real results best. So the full PointNav-v2 rules were never themselves checked against a real robot.

Confidence: medium

Known problems 5

  1. Test servers closed and the software is no longer developed

    No test set scored by the organisers is still open.13149+2

    Details

    The 2021–2023 EvalAI phases ended on 2026-01-16, the starter repository was archived on 2023-10-31, and Meta stopped active development of Habitat-Lab after v0.3.4 (2026-05-07). New results can only be self-reported on public splits.

  2. Numbers come from different datasets and splits

    ObjectNav (finding an object of a named category) numbers in papers come from different dataset versions and data splits.2163+3

    Details

    ObjectNav moved from Matterport3D (2020–2021) to HM3D-Semantics v0.1 (2022) and v0.2 (2023), with different scenes and goal categories. Outside the contest, papers report their own validation runs. Qwen-RobotNav's ObjectNav table compares its HM3D v2 numbers with other papers' HM3D v1 numbers. VLFM notes that SemExp was evaluated on a subset of Matterport3D episodes with COCO classes. Before the 2020 challenge, the ObjectNav working group described 'often-inconsistent interpretations' of the task, which the challenge rules standardised.

  3. Simulation rankings can reverse on a real robot

    In one study in a real home, the end-to-end policy (a single learned network from sensor input to actions) that did best in simulation did worst on the robot.5

    Details

    Gervet et al. (Meta AI and partners) scored ObjectNav policies on the 2022 challenge validation split and on a Hello Robot Stretch in real homes. Among 4 end-to-end variants, the one that was best in simulation (77% success) scored 0% in the real home, and the one that was worst in simulation (48%) did best in the real home (30%). Modular methods rose from 81% in simulation to 90% across six real homes. Failures differed: real errors came mostly from depth-sensor noise, while many simulated failures came from scan reconstruction errors.

  4. PointNav was solved twice

    PointNav (reaching given coordinates) scores reached their upper limit, and the task was retired in 2022.202122+3

    Details

    Under the 2019 rules, DD-PPO 'essentially solves the task' (test-standard SPL 0.9482). The HM3D paper reports that HM3D-trained PointNav agents reach 100% on the Gibson test episodes and suggests retiring that episode set. Under the harder 2020–2021 rules, the 2021 winner reached SPL 0.74 and 96% success, while an agent with perfect GPS+compass can reach at most 0.76 SPL and 99% success in that setting. The organisers considered PointNav-v2 solved and dropped it from 2022.

  5. Agents exploited wall sliding under the 2019 rules

    In 2019, agents slid along walls in ways a real robot cannot.42

    Details

    Kadian et al. found that agents trained under the 2019 settings slid along walls after collisions, which let them take paths through space a real robot cannot pass. Under those settings nearly all 9 tested models scored close to 100% success in simulation, while real-robot success varied widely; the correlation of success between simulation and reality was 0.18. From 2020 the challenge disabled sliding, removed the perfect GPS+compass and added motion noise.

Details

About

What it is
Competition123
More

Each edition page describes a yearly challenge with deadlines, prizes (2019–2021) and winners announced at CVPR workshops.

Built by
Meta AI (FAIR), Matterport31325
More

The challenge pages give organiser names but not affiliations. Co-authors of the task papers come from Georgia Tech, Simon Fraser University, Oregon State and other universities (s29, s32).

Meta AI (FAIR) · Habitat team. EvalAI host team 'FAIR A-STAR (Habitat)'; the challenge site is copyright Meta Platforms. The 2022 and 2023 citation entries list 13 and 16 organisers.16313

Matterport · Released the HM3D scenes used in 2022 and 2023 together with Facebook AI Research.25

Released
April 20191
More

Habitat Challenge 2019 opened on 2019-04-03 (EvalAI challenge 254).

Version
5 editions, from 2019 to 20231224+2
More

Five editions from 2019 to 2023. The tasks, scene datasets and rules changed in almost every edition (see items).

2019 · PointNav (reach coordinates) on Gibson scenes, RGB and RGB-D tracks. The agent had a perfect GPS+compass, and it slid along walls when it hit them. 2019-04-03 to 2019-05-18.12

2020 · PointNav-v2 on Gibson: no GPS+compass, motion and sensor noise measured on a LoCoBot robot, no sliding, LoCoBot-sized agent. New ObjectNav track on 90 Matterport3D scenes with 21 goal categories, RGB-D camera plus noiseless GPS+compass. 2020-02-24 to 2020-05-31.2

2021 · Same two tracks; the PointNav camera was tilted down. 2021-02-17 to 2021-05-31.24

2022 · ObjectNav only, on HM3D-Semantics v0.1: 120 scenes split 80/20/20 and 6 goal categories (chair, couch, potted plant, bed, toilet, tv). PointNav retired, servers left open. 2022-02-14 to 2022-08-31.16

2023 · ObjectNav and InstanceImageNav (reach the object shown in a photo) on HM3D-Semantics v0.2: 216 scenes split 145/36/35, same 6 categories, single-floor episodes, Hello Robot Stretch configuration with continuous actions allowed. 2023-03-13 to 2023-05-31.3

Last update
The last edition ended in May 2023.31314+1
More

The last edition closed on 2023-05-31. The EvalAI test servers for 2021–2023 stayed open until 2026-01-16. The newest public test-standard entry is dated 2024-05-13 (2022 ObjectNav).

Status
Closed. The last edition was in 2023. Inferred7913+3
More

No edition after 2023. The starter repository was archived on 2023-10-31, the EvalAI phases ended on 2026-01-16, and Meta stopped active development of Habitat-Lab after v0.3.4 (2026-05-07).

aihabitat.org/challenge/ redirects to the 2023 HomeRobot OVMM page and /challenge/2024/ returns HTTP 404 (checked 2026-10-10). The Habitat-Lab README now says the project 'is no longer receiving official active development or maintenance by Meta internal teams'.

Setup

Runs in
Simulation3
Simulator
Habitat-Sim37
Robot
Wheels only23
More

A simulated wheeled robot that only moves its base. In 2023 it was modelled on the Hello Robot Stretch, a mobile manipulator, but the tasks did not use the arm.

Robot model
2020–2021 PointNav matched a LoCoBot's size, camera and motion noise; ObjectNav matched an Azure Kinect camera; 2023 modelled the Hello Robot Stretch.2323
Setting
Whole home, Mixed Inferred254
More

HM3D README: 'residential, commercial, and civic spaces'. Kadian et al. describe Gibson as apartments, houses, offices, hospitals and gyms.

Tasks
3 tasks, with the goal given as a point, an object category or a photo23
More

Three tasks across editions: PointNav (reach given coordinates), ObjectNav (find any instance of a named category) and InstanceImageNav (reach the object shown in a goal photo)

Scenes
Up to 216 scanned buildings, in 20232163
More

2019–2021 PointNav: Gibson scenes with Gibson's own splits. 2020–2021 ObjectNav: 90 Matterport3D scenes. 2022: 120 HM3D-Semantics v0.1 scenes (80/20/20). 2023: 216 HM3D-Semantics v0.2 scenes (145/36/35).

Changes at test
Scene layout, New objects Inferred2163+1
More

InstanceImageNav (2023) samples goal-photo cameras independently of the agent's camera (s5, s36).

Scoring and access

Scored by
Success rate, Path efficiency2163
More

Entries were ranked by SPL. Tables also show success rate, soft SPL and distance to goal (2023 adds collisions and steps).

Score
Ranked by SPL (a success rate that also rewards short paths). Success is also shown.2163
More

Success: the agent must stop close enough to the goal. PointNav (2020–2021) required stopping within 0.36 m, twice the agent's radius. ObjectNav requires stopping within 1.0 m of any instance of the target category, at a spot from which the object could be seen by turning or tilting the camera. SPL (success weighted by path length) gives each successful episode the shortest-path length divided by the longer of the agent's path and the shortest path; failed episodes score 0; the result is averaged. A perfect, direct run scores 1.0.

In ObjectNav the shortest path is measured to the instance closest to the start, so stopping at a farther chair counts as a success but lowers SPL. From 2021 the organisers reserved the right to use other metrics when SPL differences are statistically insignificant.

Trials
Hidden test episodes: 1,000 to 2,000 per submission in 2020–2021, 1,000 in 2022–2023, within 24 hours (2020) or 48 hours (2021–2023) on an AWS p2.xlarge instance with a Tesla K80 GPU.22416+1
More

Test-challenge allowed 5 submissions per team in total; test-standard allowed up to 10 a day.

Who runs it
The organisers run the tests2163
More

Outside the contest, and since the servers closed, papers report their own runs on public validation splits (see issues.i4).

Error bars
Not reported Inferred292630
More

Leaderboards show single numbers per entry.

Leaderboard
Official, on EvalAI. It is now closed.211314
More

Official EvalAI leaderboards for each edition (challenges 254, 580, 802, 1615 and 1992). All are closed; the 2021–2023 phases ended on 2026-01-16.

Public 2023 test-challenge tables list two non-baseline ObjectNav teams and one InstanceImageNav team. Only entries that teams chose to make public appear.

Code licence
MIT789
More

habitat-challenge (archived), Habitat-Lab and Habitat-Sim are all MIT (GitHub licence fields and README).

Data licence
CC BY-NC-SA 3.0 US for the Matterport3D and Gibson episodes831
More

Episode datasets built on Matterport3D or Gibson: CC BY-NC-SA 3.0 US, plus the scene dataset's terms. HM3D-based episode sets (ObjectNav 2022–2023, InstanceImageNav): no licence statement found.

Habitat-Lab README: 'The trained models and the task datasets are considered data derived from the correspondent scene datasets.' DATASETS.md lists the HM3D episode downloads without a licence.

Asset licence
Matterport terms for academic use only253233+4
More

HM3D, HM3D-Semantics and Matterport3D scenes fall under Matterport's End User License Agreement for Academic Use: non-commercial academic use only. The licence defines models trained on the data as 'derived information' and forbids using it for non-academic purposes. Gibson scenes need a signed licence agreement.

The Matterport3D Terms of Use PDF contains the same academic-use agreement. The Gibson agreement PDF linked from Habitat-Lab returned HTTP 403 on 2026-10-10 (s28), so its terms were not read.

Access
By application Inferred253336
More

Scenes are gated. HM3D: request access from Matterport, then use an API token. Matterport3D: sign the terms of use and email them to receive a download script. Gibson: fill in a form with the licence agreement. Episode files and code are open downloads.

Commercial use
Not allowed Inferred328
More

The Matterport academic licence allows only non-commercial academic use of the scenes and of models trained on them; the episode datasets on Matterport3D and Gibson are CC BY-NC-SA. Code is MIT. Not legal advice.

Sources 37

  1. 1Habitat Challenge 2019 (PointGoal, RGB and RGB-D tracks; results)Official site · 2019 · checked 10 Oct 2026
  2. 2Habitat Challenge 2020 (PointNav and ObjectNav; 'New in 2020'; results)Official site · 2020 · checked 10 Oct 2026
  3. 3Habitat Navigation Challenge 2023 (ObjectNav and InstanceImageNav on HM3D-Semantics v0.2)Official site · 2023 · checked 10 Oct 2026
  4. 4Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? (Kadian et al., RA-L 2020; full text v2)Paper · Dec 2019 · checked 10 Oct 2026
  5. 5Navigating to Objects in the Real World (Gervet et al.; Table 1, Figure 3)Paper · Dec 2022 · checked 10 Oct 2026
  6. 6Habitat-Matterport 3D Semantics Dataset (Section 4.4, Table 5)Paper · Oct 2022 · checked 10 Oct 2026
  7. 7facebookresearch/habitat-challenge (starter code; MIT; archived 2023-10-31)Repository · 24 Apr 2023 · checked 10 Oct 2026
  8. 8Habitat-Lab README (licence of task datasets; support notice after v0.3.4)Repository · 7 May 2026 · checked 10 Oct 2026
  9. 9GitHub API: facebookresearch/habitat-challenge, habitat-lab and habitat-sim (stars, archive date)Index · 10 Oct 2026 · checked 10 Oct 2026
  10. 10HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal NavigationPaper · Sep 2024 · checked 10 Oct 2026
  11. 11MultiON: Benchmarking Semantic Map Memory using Multi-Object NavigationPaper · Dec 2020 · checked 10 Oct 2026
  12. 12NeurIPS 2023 HomeRobot Open Vocabulary Mobile Manipulation (OVMM) ChallengeOfficial site · 2023 · checked 10 Oct 2026
  13. 13EvalAI API: Habitat Navigation Challenge 2023 phases (end date 2026-01-16, inactive)Leaderboard · 16 Jan 2026 · checked 10 Oct 2026
  14. 14EvalAI API: Habitat Challenge 2022 phases (end date 2026-01-16, inactive)Leaderboard · 16 Jan 2026 · checked 10 Oct 2026
  15. 15Habitat-Lab releases (v0.3.4 on 2026-05-07)Repository · 7 May 2026 · checked 10 Oct 2026
  16. 16Habitat Challenge 2022 (ObjectNav on HM3D-Semantics v0.1; results)Official site · 2022 · checked 10 Oct 2026
  17. 17Qwen-RobotNav Technical Report (Table 4: ObjectNav on MP3D and HM3D)Paper · Jun 2026 · checked 10 Oct 2026
  18. 18VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (Table I)Paper · Dec 2023 · checked 10 Oct 2026
  19. 19ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to ObjectsPaper · Jun 2020 · checked 10 Oct 2026
  20. 20DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion FramesPaper · Nov 2019 · checked 10 Oct 2026
  21. 21EvalAI leaderboard, Habitat Challenge 2019, PointNav RGB-D test-standard (phase split 839)Leaderboard · 2019 · checked 10 Oct 2026
  22. 22Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIPaper · Sep 2021 · checked 10 Oct 2026
  23. 23Retrospectives on the Embodied AI Workshop (Section 3.1)Paper · Oct 2022 · checked 10 Oct 2026
  24. 24Habitat Challenge 2021 (PointNav and ObjectNav; results)Official site · 2021 · checked 10 Oct 2026
  25. 25Habitat-Matterport 3D Research Dataset (HM3D) READMERepository · Mar 2023 · checked 10 Oct 2026
  26. 26EvalAI leaderboard, Habitat Challenge 2022, ObjectNav test-standard (phase split 3899)Leaderboard · 2022 · checked 10 Oct 2026
  27. 27AI Habitat challenge index (redirects to the 2023 HomeRobot OVMM page; /challenge/2024/ returns 404)Official site · 10 Oct 2026 · checked 10 Oct 2026
  28. 28Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object InstancesPaper · Nov 2022 · checked 10 Oct 2026
  29. 29EvalAI leaderboard, Habitat Challenge 2020, PointNav test-standard (phase split 1631)Leaderboard · 2020 · checked 10 Oct 2026
  30. 30EvalAI leaderboard, Habitat Navigation Challenge 2023, ObjectNav test-challenge (phase split 4705)Leaderboard · 2023 · checked 10 Oct 2026
  31. 31Habitat-Lab DATASETS.md (PointNav, ObjectNav and InstanceImageNav episode downloads)Repository · 2026 · checked 10 Oct 2026
  32. 32Matterport End User License Agreement for Academic Use of Model DataOfficial site · unknown · checked 10 Oct 2026
  33. 33Matterport3D Terms of Use (PDF linked from Habitat-Lab and VLN-CE)Official site · unknown · checked 10 Oct 2026
  34. 34HM3D-Semantics dataset pageOfficial site · 2023 · checked 10 Oct 2026
  35. 35HM3D dataset pageOfficial site · 2021 · checked 10 Oct 2026
  36. 36Gibson Database of Spaces download notes (licence agreement via form)Repository · 2018 · checked 10 Oct 2026
  37. 37Gibson Terms of Use agreement PDF linked from Habitat-Lab (returned HTTP 403 on 2026-10-10)Official site · Jun 2018 · checked 10 Oct 2026
Where we searched for missing information

sim_to_real (independent replication): Web searches on 2026-10-10 for sim-to-real correlation of Habitat PointNav and ObjectNav; Retrospectives on the Embodied AI Workshop; Truong et al. 2022 'Rethinking Sim2Real' (arXiv 2207.10821, same team, own PointNav settings with legged robots, abstract read only); VLFM (real Spot deployment, no paired numbers). No independent paired study found.

license_data (HM3D episode sets): Habitat-Lab README and DATASETS.md, HM3D README, HM3D-Semantics page, 2022 and 2023 challenge pages. No licence statement for the HM3D-based ObjectNav and InstanceImageNav episode files.

license_assets (Gibson): Gibson agreement PDF linked from Habitat-Lab (HTTP 403 on 2026-10-10); GibsonEnv data README (licence via a Google form).

2024 and later editions: aihabitat.org/challenge/2024/ and /2025/ (HTTP 404); challenge menu on aihabitat.org (lists 2019–2023 navigation and 2023 OVMM); EvalAI challenge metadata.

2023 participation: 2023 challenge page (no results section), EvalAI public leaderboards for phases 4704, 4705, 4707 and 4708.

citations (ObjectNav Revisited, Sim2Real Predictivity): Semantic Scholar API; repeated HTTP 429 rate-limit responses on 2026-10-10. Counts are added only where the API answered.

Change history

  1. Created as a basic entry: identity facts checked at primary sources (phase 1 re-verification).
  2. Full entry. Re-checked all basic facts. Corrections: the 2019 PointNav winner 'Arnold' scored SPL 0.948 on test-challenge (the later test-standard best is DD-PPO, 0.9482); the Gervet et al. SRCC values are per-episode correlations for single policies, not cross-policy correlations; the ObjectNav sim-vs-real headline (77% to 23%) compares the best simulated end-to-end variant with the variant run at scale, which scored 48% in simulation. Added per-edition results from EvalAI, participation counts, licences for HM3D and Matterport3D (non-commercial, covering trained models), validity entries, and five issues.
  3. Published as a full entry.