Point Cloud as a Foreign Language for Multi-modal Large Language Model

Sneha Paul, Zachary Patterson, Nizar BouguilaarXiv 2026 · DOI · arXiv

Abstract

Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens—treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency and robustness to input resolution variations.

Introduction: The Encoder Bottleneck

Most current 3D-capable AI models follow an "encoder-dependent" paradigm. They utilize a pre-trained 3D encoder, such as Point-BERT, to extract features which are then projected into the embedding space of a Large Language Model (LLM). While functional, this approach introduces three major challenges:

  1. Semantic Misalignment: 3D encoders are typically trained for geometric classification, not linguistic reasoning, leading to a "language barrier" between the vision and text components.
  2. Resolution Brittleness: These encoders often require fixed input sizes (e.g., exactly 8,192 points), making them fragile when dealing with real-world data of varying densities.
  3. Computational Overhead: Running a massive 3D encoder before the LLM adds significant latency and memory costs.

The SAGE Methodology: 3D as a Foreign Language

The core innovation of SAGE (Spatial-Aware GEnnerative model) is its shift away from heavy encoders toward a lightweight 3D tokenizer. This allows the LLM to "read" point clouds directly, much like it reads text in a foreign language.

The SAGE architecture treats 3D data as a sequence of discrete tokens, similar to text, allowing for direct processing by the LLM.

The tokenization process involves Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) to group local geometric structures. These structures are then mapped through Vector Quantization (VQ) into a finite vocabulary of discrete 3D tokens. By extending the LLM’s dictionary to include these 3D primitives, the model can process spatial data natively within its transformer blocks.

Training and Preference Optimization

SAGE is trained in three distinct stages to ensure robust alignment:

  • Tokenizer Warm-up: Initial training of the tokenizer and the first few LLM layers on 3D captioning tasks.
  • Instruction Tuning: Fine-tuning the entire system to follow complex, multi-turn 3D-related commands.
  • Preference Optimization: Utilizing Group Relative Policy Optimization (GRPO). Because 3D descriptions are often subjective, the authors developed a reward function based on semantic alignment (using Sentence-BERT) and length regularization to ensure responses are accurate, descriptive, and concise.

Key Results and Efficiency

SAGE was evaluated on diverse benchmarks including Objaverse, MM-Vet, and ScanQA. The results indicate a significant leap in both performance and efficiency. SAGE-7B achieved a GPT-4 evaluation score of 50.98 on Objaverse captioning, surpassing the previous state-of-the-art of 48.15.

SAGE demonstrates superior performance and robustness across varying input resolutions compared to the encoder-based PointLLM.

Beyond accuracy, SAGE offers a 2.3x speedup in inference latency (100ms vs 239ms for PointLLM) and handles more than double the throughput. Crucially, as shown in the figure above, SAGE maintains high performance across different point cloud resolutions (2K to 8K), whereas encoder-based models often see performance degradation or require costly padding to fit fixed encoder inputs.

Conclusion

SAGE demonstrates that specialized 3D encoders are not a prerequisite for high-quality 3D understanding. By treating point clouds as a sequence of tokens, SAGE achieves a more seamless integration of spatial and linguistic modalities. This work paves the way for truly unified multimodal models that can process diverse data types—images, 3D, and audio—within a single, efficient, encoder-free architecture.

Code and models are available at github.com/snehaputul/SAGE3D.

Summary updated

Abstract

Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens--treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Furthermore, to enhance the model's reasoning capability on complex 3D tasks, we propose a preference optimization training strategy with a semantic alignment-based reward, specifically designed for open-ended 3D question answering where responses are descriptive. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency, generalization across LLM backbones, and robustness to input resolution variations. Code is available at: github.com/snehaputul/SAGE3D.
BibTeX
@misc{paul2026point,
  author = {Sneha Paul and Zachary Patterson and Nizar Bouguila},
  title = {{Point Cloud as a Foreign Language for Multi-modal Large Language Model}},
  year = {2026},
  eprint = {2603.09173},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.09173},
}

Loading related content...

Figure