Abstract
This paper introduces SMArT (Shallow and Memory-Aware Transformer), a novel architecture designed for efficient image captioning in resource-constrained robotic environments. By utilizing a shallow 2-layer structure augmented with persistent memory vectors, SMArT achieves state-of-the-art performance on the MS COCO dataset while significantly reducing prediction latency compared to deeper Transformer models. Our results demonstrate that model depth is not a prerequisite for high-quality captioning and that memory-augmented shallow layers can effectively bridge the gap between efficiency and accuracy for autonomous agents.
Introduction
For autonomous robots to be truly integrated into human environments, they must be "trustworthy," which requires the ability to explain their perceptions and actions in natural language. While image captioning has advanced rapidly, recent progress has largely relied on increasing model depth—often employing Transformers with six or more layers. This trend toward complexity poses a significant challenge for robotics, where agents typically operate on embedded hardware with limited power and memory.
Existing high-performance models are often too computationally expensive for real-time robotic applications. This research addresses the gap by proposing a "shallow" model that maintains high accuracy while minimizing computational overhead, specifically tailored for the needs of real-time robotic explainability.
The SMArT Architecture
The SMArT model departs from the standard "deeper is better" paradigm by employing a streamlined 2-layer encoder and 2-layer decoder. To compensate for the reduced depth, the authors introduce a Memory-Augmented Region Encoder.
In standard self-attention mechanisms, image regions interact to establish context. SMArT extends this by incorporating persistent memory vectors—learnable parameters that act as slots for global dataset knowledge. These vectors allow the model to leverage a-priori knowledge learned during training, effectively providing a "shortcut" for encoding visual features without requiring the abstraction depth of a 6-layer model. The training process involves an initial cross-entropy loss stage followed by Reinforcement Learning (Self-Critical Sequence Training) to optimize for the CIDEr metric.
Experimental Results
Performance on MS COCO
Evaluated on the Microsoft COCO dataset, SMArT achieved a CIDEr score of 129.7. This performance notably outstrips more complex architectures, such as the 6-layer ORT (128.3) and the Up-Down model (120.1). Ablation studies revealed that a 2-layer Transformer actually performs better for image captioning than a 6-layer counterpart, suggesting that the visual-to-lingual mapping task does not require the same depth as pure machine translation.
Robotic Utility and Coverage
To test real-world robotic applicability, the model was evaluated on the ACVR Robotic Vision Challenge dataset, which features simulated indoor scenes. SMArT achieved a 0.76 coverage score (identifying objects occupying at least 1% of the image), indicating its reliability in naming relevant objects within a robot's field of view.
Computational Efficiency
A primary goal of SMArT is reducing latency for onboard deployment. Testing on an NVIDIA 1080Ti confirmed that SMArT maintains significantly lower prediction latency than deeper competitors.
As shown in the latency analysis, SMArT (with $m=40$ memory slots) scales efficiently with batch size, making it a viable candidate for real-time processing on robotic platforms where deeper models like ORT become prohibitive.
Conclusion and Future Work
SMArT demonstrates that shallow, memory-aware Transformers can achieve state-of-the-art image captioning performance with superior efficiency. While currently validated on simulated data, future work will focus on deploying SMArT in physical, unstructured environments and integrating dynamic memory that can update as a robot explores new spaces. Ultimately, the goal is to bridge the gap between vision, language, and action, allowing robots to explain not just what they see, but what they are doing.
Summary updated