Ego2Sim

Teleoperated Spot in Isaac Sim, in an outdoor scene built entirely by the Ego2Sim pipeline.

Abstract

Vision-based navigation and locomotion policies are increasingly trained in digital twins of real environments. However, a training environment with both the visual appearance and the contact geometry of a scene often requires a dedicated scanner or a manual registration step. Egocentric glasses offer a lightweight alternative: they capture a scene hands-free as the wearer walks through it, and recordings from such glasses already serve as human demonstrations for robot learning. In this paper, we present a real-to-sim pipeline that builds a realistic and contact-enabled digital twin in Isaac Sim from a single walk recorded with Meta Aria Gen 2 glasses. The resulting hybrid scene pairs a Gaussian splat for appearance with a collision mesh derived from the glasses' own SLAM points. Compared against same-day iPad LiDAR scans, the collision mesh achieves an F-score of 0.84–0.94 at 10 cm within 1 m of the ground, taking 51–160× less time to build than a multi-view stereo mesh (MVS).