Think-Program-reCtify: 3D Situated Reasoning with Large Language Models

Qingrong He, Kejun Lin, Shizhe Chen, Anwen Hu, Qin JinarXiv 2024 · DOI · arXiv

Introduction

3D Situated Reasoning (3D-SR) is a complex task that requires an autonomous agent to understand its environment from an egocentric perspective. Unlike standard visual question answering, 3D-SR demands that an agent comprehends its specific position and orientation within a 3D space to answer questions like, "Which direction should I go if I want to iron my clothes?"

Figure 1: An example of the 3D Situated Reasoning task, highlighting the multi-faceted challenges including spatial reasoning, navigation, and common-sense knowledge.

Existing approaches typically rely on end-to-end "black box" models that require massive amounts of labeled 3D-text data or language-only models that use sparse text captions to represent 3D scenes. These methods often struggle with multi-step logic and lack interpretability. To address these issues, the authors introduce LLM-TPC, a training-free framework that leverages the reasoning power of Large Language Models (LLMs) through a structured execution loop.

Methodology: The Think-Program-reCtify Loop

LLM-TPC treats the LLM as a high-level controller that orchestrates specialized 3D vision tools. The framework operates through a three-phase iterative loop:

  1. Think: The LLM decomposes a complex situated question into a sequence of logical natural language steps.
  2. Program: These steps are translated into a Python program that calls a custom API for 3D perception. This API includes modules like Mask3D for segmentation, OpenShape for object classification, and spatial modules for calculating relative positions.
  3. reCtify: If the program fails during execution due to syntax or logic errors, the LLM receives the error feedback and enters a self-reflection phase to debug and adjust its plan.
Figure 5: Comparison of (a) End-to-end methods, (b) Language-only methods, and (c) the proposed LLM-TPC framework which utilizes a 3D visual perception layer and a Think-Program-Rectify reasoning loop.

To further enhance performance, the system uses an Ensemble approach, merging the logical reasoning of LLM-TPC with the top predictions from traditional end-to-end models like 3D-VisTA.

Experimental Evaluation

The framework was evaluated on the SQA3D benchmark, which contains over 33,000 Q&A pairs across 650 ScanNet scenes. LLM-TPC demonstrated significant improvements over existing supervised models:

  • State-of-the-Art Performance: Using ground-truth segmentation, LLM-TPC achieved 56.92% accuracy, surpassing the supervised LEO model (53.25%).
  • Skill-Specific Gains: The model excelled in reasoning-intensive categories, specifically Navigation (77.04% vs. 40.74% for 3D-VisTA) and Calculation (65.28% vs. 54.63%).
  • Interpretability: By generating executable code and natural language plans, the model provides a transparent reasoning chain that is easier to debug than standard neural networks.

Key Insights and Limitations

A major finding of the study is the perception bottleneck. While the LLM's reasoning is robust, the system's overall accuracy is heavily dependent on the quality of 3D segmentation. When provided with perfect object segments, performance jumps significantly, suggesting that future improvements in 3D-SR will likely stem from more advanced 3D foundation models for perception.

Additionally, the "Rectify" phase proved essential; without self-correction, failure rates for complex programs exceeded 20%. This highlights the importance of closed-loop feedback in embodied AI systems. Future work will focus on integrating more flexible perception tools and improving the information density of the feedback provided to the LLM during the rectification phase.

Summary updated

Abstract

This work addresses the 3D situated reasoning task which aims to answer questions given egocentric observations in a 3D environment. The task remains challenging as it requires comprehensive 3D perception and complex reasoning skills. End-to-end models trained on supervised data for 3D situated reasoning suffer from data scarcity and generalization ability. Inspired by the recent success of leveraging large language models (LLMs) for visual reasoning, we propose LLM-TPC, a novel framework that leverages the planning, tool usage, and reflection capabilities of LLMs through a ThinkProgram-reCtify loop. The Think phase first decomposes the compositional question into a sequence of steps, and then the Program phase grounds each step to a piece of code and calls carefully designed 3D visual perception modules. Finally, the Rectify phase adjusts the plan and code if the program fails to execute. Experiments and analysis on the SQA3D benchmark demonstrate the effectiveness, interpretability and robustness of our method. Our code is publicly available at https://qingrongh.github.io/LLM-TPC/.
BibTeX
@misc{he2024think,
  author = {Qingrong He and Kejun Lin and Shizhe Chen and Anwen Hu and Qin Jin},
  title = {{Think-Program-reCtify: 3D Situated Reasoning with Large Language Models}},
  year = {2024},
  eprint = {2404.14705},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.14705},
}
Figure