Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments

Muzhi Han*, Zeyu Zhang*, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu and Hangxin Liu

UCLA Center for Vision, Cognition, Learning, and Autonomy (VCLA)

Abstract

In this paper, we rethink the problem of scene reconstruction from an embodied agent's perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide actionable information for simulating interactions with agents. Here, we address this challenging problem by reconstructing an interactive scene using RGB-D data stream, which captures (i) the semantics and geometry of objects and layouts by a 3D volumetric panoptic mapping module, and (ii) object affordance and contextual relations by reasoning over physical common sense among objects, organized by a graph-based scene representation. Crucially, this reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models for finer-grained robot interactions. In the experiments, we demonstrate that (i) our panoptic mapping module outperforms previous state-of-the-art methods, (ii) a high-performant physical reasoning procedure that matches, aligns, and replaces objects' meshes with best-fitted CAD models, and (iii) reconstructed scenes are physically plausible and naturally afford actionable interactions; without any manual labeling, they are seamlessly imported to ROS-based simulators and virtual environments for complex robot task executions.

Bibtex

@inproceedings{han2021reconstructing,

title={Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments},

author={Han, Muzhi and Zhang, Zeyu and Jiao, Ziyuan and Xie, Xu and Zhu, Yixin and Zhu, Song-Chun and Liu, Hangxin},

booktitle={ICRA},

year={2021}

}

ICRA Presentation

presentation_1080p.mp4

Video Demos

camera_ready_1080p.mp4

Overview

supp_1080p.mp4

Additional Results

Center for Vision, Cognition, Learning, and Autonomy (VCLA@UCLA)

@2021

Page updated

Google Sites

Report abuse

	
		OSZAR »