ROOT
VLM based System for Indoor Scene Understanding and Beyond

Yonghui Wang1,2, Shi-Yong Chen2, Zhenxing Zhou2, Siyi Li2, Haoran Li1,2, Wengang Zhou1, Houqiang Li1
1University of Science and Technology of China (USTC) 2Game AI Center, Tencent IEG
description Paper article arXiv code Code emoji_emotions Hugging Face

Scene Attribute Analysis

  • Object Detection
  • Bounding Boxes
  • Instance Masks
  • Depth Maps
  • Point Clouds

Spatial Relationship

  • Hierarchical Scene Graphs
  • Object Distance Estimation

Abstract

Recently, Vision Language Models (VLMs) have experienced significant advancements, yet these models still face challenges in spatial hierarchical reasoning within indoor scenes.

In this study, we introduce ROOT, a VLM-based system designed to enhance the analysis of indoor scenes. Specifically, we first develop an iterative object perception algorithm using GPT-4V to detect object entities within indoor scenes. This is followed by employing vision foundation models to acquire additional meta-information about the scene, such as bounding boxes. Building on this foundational data, we propose a specialized VLM, SceneVLM, which is capable of generating spatial hierarchical scene graphs and providing distance information for objects within indoor environments. This information enhances our understanding of the spatial arrangement of indoor scenes. To train our SceneVLM, we collect over 610,000 images from various public indoor datasets and implement a scene data generation pipeline with a semi-automated technique to establish relationships and estimate distances among indoor objects. By utilizing this enriched data, we conduct various training recipes and finish SceneVLM.

Our experiments demonstrate that ROOT facilitates indoor scene understanding and proves effective in diverse downstream applications, such as 3D scene generation and embodied AI. The code will be released at https://github.com/harrytea/ROOT.

Approach

ROOT Approach
Overview of our ROOT system. Given an indoor scene image, we first employ GPT-4V to iteratively detect objects. Then, we utilize vision foundation models to obtain meta-information such as bounding boxes. Finally, our SceneVLM generates spatial hierarchical scene graphs and provides distance information, enabling comprehensive indoor scene understanding.

Hierarchical Scene Graph Visualization

Bedroom Vanity

Modern Dining Room

Modern Living Room

Modern Living Room

Modern Living Room

Dining Room

Modern Living Room

Video Demo

Potential Applications

Application 1
Scene VQA
Application 2
Smart Placement
Application 5
Scene Generation
Application 3
Game AI
Application 4
Embodied AI

Below we showcase a detailed example of Scene Generation with Holodeck. For other applications, please refer to our paper.

ROOT

ROOT

Spatial Relationships
Holodeck

Holodeck

ROOT's scene understanding capabilities can potentially work together with Holodeck for scene generation. By providing spatial relationship information and object placement suggestions, ROOT aims to assist Holodeck in generating indoor environments.

Holodeck Generated Scene 1
A daycare with playrooms connected to nap rooms and a kitchen
Holodeck Generated Scene 2
A fitness center with a gym area connected to locker rooms and showers

BibTeX

@article{wang2024root,
  title={ROOT: VLM based System for Indoor Scene Understanding and Beyond},
  author={Wang, Yonghui and Chen, Shi-Yong and Zhou, Zhenxing and Li, Siyi and Li, Haoran and Zhou, Wengang and Li, Houqiang},
  journal={arXiv preprint arXiv:2411.15714},
  year={2024}
}