M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer

Yanxin Zhang, Liang He, Zeyi Kang, Zuheng Ming, Kaixing Zhao

TL;DR

M3ET is a lightweight multimodal framework that combines the strengths of Transformers with the efficiency of Mamba (State Space Models) to enable real-time vision-language learning on resource-constrained robots. By reducing parameter counts by over 66% and boosting inference speeds by 2.3x, it achieves competitive performance on tasks like Visual Question Answering (VQA) while remaining small enough for edge deployment.

What Problem Does This Paper Address?

In the world of robotics, understanding a complex environment requires fusing data from multiple "modalities"—think RGB images, depth maps, and natural language instructions. While large-scale models like CLIP or ViT-based Masked Autoencoders (MAEs) are powerful, they are often too computationally "heavy" for the limited hardware found on mobile robots.

Furthermore, these models often struggle when data is incomplete (modality loss) or fail to deeply integrate textual semantics with visual features. Current methods typically stack textual embeddings onto visual features in a way that doesn't fully exploit the rich context language provides. The authors aim to bridge this gap by creating a model that is both semantically rich and physically lean.

Methods

The core innovation of M3ET (Multi-Modal Mamba-Enhanced Transformer) is its hybrid hierarchical architecture.

  1. Mamba Integration: Instead of relying solely on standard Transformer blocks—which suffer from quadratic computational complexity—the authors incorporate Mamba modules. Mamba is based on State Space Models (SSMs) and offers linear scaling with sequence length, making it much more efficient for processing the long sequences of tokens generated by multimodal inputs.
  2. Cross-Modal Encoder: The encoder uses alternating Transformer and Mamba blocks. This allows the model to capture local features via self-attention while using Mamba to efficiently model long-range dependencies across different modalities.
  3. Semantic-Based Adaptive Attention: The model uses a dynamic fusion mechanism to align visual (RGB, depth) and linguistic features. It stores modality-specific embeddings in a parameter dictionary, ensuring the unique characteristics of each data type are preserved during fusion.
  4. Multi-modal Masking Strategy: During training, the model uses a Dirichlet distribution to sample masking ratios across modalities. Notably, it employs "whole-sentence masking" for text (80% probability) to force the model to reconstruct semantic meaning from visual cues alone.

Datasets

The authors evaluated M3ET across four diverse benchmarks to test reconstruction, perception, and reasoning:

  • ImageNet-1K: Used for general image classification and reconstruction.
  • NYUv2: Focused on depth estimation and semantic segmentation in indoor environments.
  • VQAv2: The standard benchmark for Visual Question Answering.
  • OpenEQA: A challenging "Embodied Question Answering" dataset featuring over 1,600 open-domain questions about real-world environments.

Key Insights and Ideas

  • The Efficiency-Performance Sweet Spot: The most significant "aha" moment is that you don't need a massive Transformer to achieve high accuracy. By replacing parts of the Transformer layers (7.09M parameters each) with Mamba components (only 100.67K parameters each), the authors slashed the total parameter count from ~196M to ~65M without a significant drop in VQA accuracy.
  • Language as a Scaffold: The ablation studies show that the text modality is crucial even for visual tasks. Removing text reduced the Peak Signal-to-Noise Ratio (PSNR) in image reconstruction, proving that high-level linguistic concepts actually help the model "fill in the blanks" of low-level sensory data.
  • Mamba as a Compressor: In the decoder, a MambaBlock is used to compress spatial and textual representations from 256 dimensions down to 64 and back again. This bottleneck forces the model to learn the most essential cross-modal features.

Key Results

  • Speed & Memory: M3ET achieved a 2.3x increase in pre-training inference speed and a 73.2% reduction in memory consumption compared to the baseline MultiMAE.
  • Parameter Count: The model is significantly leaner, with a 66.63% reduction in parameters (65.39M vs. 195.97M).
  • VQA Accuracy: On the VQAv2 task, M3ET reached 74.18% accuracy, outperforming established models like ViLBERT (70.55%) and UNITER (68.19%).
  • Reconstruction: On ImageNet-1K, it achieved 83.6% top-1 accuracy, slightly edging out MultiMAE (83.3%) despite being much smaller.

Limitations and Future Work

While M3ET excels at efficiency, it still struggles with high-level reasoning in the Embodied Question Answering (EQA) task, where it achieved only 14.29% accuracy. This is significantly lower than state-of-the-art models based on massive LLMs (like GPT-4), highlighting a trade-off between "edge-readiness" and "deep reasoning."

The authors plan to address this by:

  • Improving the model's ability to handle complex temporal dependencies (crucial for video-based robotics).
  • Deploying and testing the model on physical hardware like NVIDIA Jetson or ROS-based agents to validate real-world latency and power consumption.

Summary updated

Abstract

In recent years, multimodal learning has become essential in robotic vision and information fusion, especially for understanding human behavior in complex environments. However, current methods struggle to fully leverage the textual modality, relying on supervised pretrained models, which limits semantic extraction in unsupervised robotic environments, particularly with significant modality loss. These methods also tend to be computationally intensive, leading to high resource consumption in real-world applications. To address these challenges, we propose the Multi Modal Mamba Enhanced Transformer (M3ET), a lightweight model designed for efficient multimodal learning, particularly on mobile platforms. By incorporating the Mamba module and a semantic-based adaptive attention mechanism, M3ET optimizes feature fusion, alignment, and modality reconstruction. Our experiments show that M3ET improves cross-task performance, with a 2.3 times increase in pretraining inference speed. In particular, the core VQA task accuracy of M3ET remains at 0.74, while the model's parameter count is reduced by 0.67. Although performance on the EQA task is limited, M3ET's lightweight design makes it well suited for deployment on resource-constrained robotic platforms.
BibTeX
@misc{zhang2025m,
  author = {Yanxin Zhang and Liang He and Zeyi Kang and Zuheng Ming and Kaixing Zhao},
  title = {{M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer}},
  year = {2025},
  eprint = {2509.18005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.18005},
}
Figure