DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Tasks

Jiayi Song, Rui Wan, Lipeng Ma, Weidong Yang, Qingyuan Zhou, Yixuan Li, Ben FeiarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces DeepThink3D, a novel framework that significantly enhances Large Language Models' (LLMs) ability to perform complex reasoning in 3D environments. By combining supervised fine-tuning (SFT) with Direct Preference Optimization (DPO) and augmenting the training data with more complex questions, DeepThink3D achieves state-of-the-art performance in 3D Situated Reasoning tasks, improving both accuracy and code executability.

What Problem Does This Paper Address?

Embodied AI agents need to understand and reason about the 3D physical world to perform complex tasks like navigation and interaction. 3D Situated Reasoning (3D-SR) aims to equip agents with this capability, requiring them to integrate 3D perception, natural language understanding, and spatial reasoning. While recent approaches have leveraged LLMs with tool-use (APIs) for 3D-SR, they face challenges: (1) weak reasoning ability leading to unstructured and incorrect reasoning paths, and (2) limited code executability due to generated code errors. Existing datasets often feature simple questions, leading to short reasoning chains and hindering the development of more robust models. This paper addresses the need for LLMs that can perform more complex, interpretable, and reliable reasoning in 3D environments.

Methods

DeepThink3D tackles the identified challenges through a multi-stage approach:

  • Structured Tool Use: LLMs are equipped with a set of 3D visual perception modules and APIs (Scene Description, Object Filtering, Object Querying by Relation, Object Information Querying) to interact with 3D scenes. Complex tasks are decomposed into sequential reasoning steps, translated into executable code.
  • Reasoning-Oriented Supervised Fine-Tuning (SFT): The model is trained on successful reasoning chains and generated code from an iterative process. This teaches the LLM to generate interpretable, step-by-step solutions and understand the logic of reasoning.
  • Execution-Oriented Direct Preference Optimization (DPO): This stage refines the model's ability to generate logically coherent and executable toolchains. It uses pairs of correct and incorrect code samples (including those that fail to run or produce wrong answers) to optimize the model towards generating more reliable and effective code.
  • Data Augmentation: To address the simplicity of existing datasets, the authors use an LLM to generate more complex, multi-step questions from the original SQA3D dataset. This creates a more challenging training environment, pushing the model to develop deeper reasoning capabilities.

Datasets

  • SQA3D [22]: This dataset serves as the foundation for training and evaluation. It consists of 650 scenes from ScanNet, with scene recognition and question-answer pairs. The authors use 26,623 training and 3,519 test samples.
  • Augmented SQA3D: The authors augment the original SQA3D dataset by using an LLM to generate more complex questions, increasing the reasoning difficulty and diversity.

Key Insights and Ideas

  • Decomposition is Key: Breaking down complex 3D reasoning tasks into a sequence of API calls (toolchains) is crucial for interpretability and controllability.
  • Beyond Simple Code Generation: Simply generating code isn't enough; the code must be executable and logically sound. The combination of SFT and DPO is vital for achieving this.
  • Data Augmentation for Complexity: Simple datasets limit LLM reasoning. Synthesizing more complex questions is essential for training models that can handle nuanced 3D scenarios.
  • Iterative Refinement Pays Off: The iterative process of generating code, observing results, and correcting errors (both in code and reasoning) significantly improves model performance.

Key Results

  • State-of-the-Art Performance: DeepThink3D achieves the highest accuracy (62.11%) on the SQA3D dataset compared to existing methods like ScanQA, 3D-VisTA, 3D-LLM, LEO, and LLM-TPC.
  • Improved Executability: Over 75% of correctly answered questions are solved with code that returns the correct answer on the first execution, highlighting the effectiveness of SFT and DPO.
  • Reduced Communication Rounds: DeepThink3D achieves correct answers in the first generation round for 82.3% of cases, a significant improvement over LLM-TPC.
  • Robustness to Errors: The model demonstrates strong self-correction capabilities, handling API errors, logic errors, and task understanding errors effectively.

Limitations and Future Work

  • Dependence on Perception Modules: The framework's performance is influenced by the accuracy of underlying 3D visual perception modules (e.g., object segmentation). Current models, while good, are not perfect and can introduce noise. Future work could focus on improving these modules or developing methods more robust to their imperfections.
  • LLM Limitations: LLMs can still deviate from instructions or get trapped in local fixes, leading to unpredictability. Further improvements in LLM capabilities are needed for more coherent and reliable reasoning.
  • Iterative Refinement: While DeepThink3D shows a small improvement in requiring up to six rounds for some cases, indicating persistent refinement, this also highlights that a small percentage of tasks still require extensive iteration. Future work could aim to further reduce this.

Summary updated

Abstract

This work enhances the ability of large language models (LLMs) to perform complex reasoning in 3D scenes. Recent work has addressed the 3D situated reasoning task by invoking tool usage through large language models. Large language models call tools via APIs and integrate the generated programs through a chain of thought to solve problems based on the program results. However, due to the simplicity of the questions in the dataset, the generated program reasoning chains are relatively short. To solve this main challenge, in this paper, we introduce DeepThink3D to enhance the tool usage of LLMs in complex 3D situated reasoning tasks. Our work proposes a combinatorial and iterative evolutionary approach on the SQA3D benchmark to generate more complex questions. Building on this foundation, we fine-tune the large language model to make it more proficient in using 3D tools. By employing Direct Preference Optimization (DPO), we directly optimize the toolchain strategies generated by models, thereby enhancing their accuracy in complex tasks.
BibTeX
@misc{song2025deepthink,
  author = {Jiayi Song and Rui Wan and Lipeng Ma and Weidong Yang and Qingyuan Zhou and Yixuan Li and Ben Fei},
  title = {{DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Tasks}},
  year = {2025},
  eprint = {2508.15548},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.15548},
}
Figure