TL;DR
This paper introduces the Transformer Dual-Stream Information (TDSI) model for generating descriptive captions for maritime images. By fusing detailed grid-level visual features with semantic segmentation information, TDSI significantly improves captioning accuracy in complex maritime environments, outperforming existing state-of-the-art methods.
What Problem Does This Paper Address?
Intelligent ships require robust environmental perception to ensure navigation safety and enhance operational efficiency. Image captioning is a key technology for this, allowing ships to "understand" their surroundings by converting visual data into textual descriptions. However, existing general-purpose image captioning models often falter in the unique and complex maritime environment. This is due to limitations in extracting sufficient visual detail and accurately modeling the semantics of maritime scenes. The authors aim to bridge this gap by developing a model specifically tailored for maritime image captioning.
Methods
The core of this work is the Transformer Dual-Stream Information (TDSI) model. It employs a dual-encoder architecture that processes visual information from two distinct streams:
- Grid Features: Extracted using a pre-trained Swin-Transformer, which captures fine-grained visual details at a grid level.
- Segmentation Features: Obtained using the SegFormer model, which provides semantic understanding of different regions within the image (e.g., identifying water, ships, buoys, coastline).
These two feature streams are independently encoded by separate Transformer encoders. A feature fusion module then implicitly combines these encoded features. Crucially, a cross-attention mechanism is integrated into the decoder to further enhance the interaction and fusion between the grid and segmentation features. This allows the model to leverage both detailed visual textures and semantic scene understanding for caption generation. The decoder then uses these fused features to generate descriptive captions.
Datasets
The authors highlight the scarcity of suitable datasets for maritime image captioning and semantic segmentation. To address this, they constructed two new datasets:
- Maritime Semantic Segmentation dataset (MSS): Contains 2405 images categorized into seven classes, collected from real sailing videos.
- Maritime Image Caption dataset (MIC): Consists of 890 images, each with five Chinese descriptive statements.
These datasets were split into training, validation, and testing sets in an 8:1:1 ratio.
Key Insights and Ideas
- Dual-Stream Fusion is Crucial for Maritime Scenes: The maritime environment presents unique challenges (e.g., vast open water, varying visibility, specific objects like buoys and ships). Relying solely on general visual features is insufficient. Combining detailed visual textures (grid features) with semantic scene understanding (segmentation features) is key to accurate captioning.
- SegFormer's Semantic Power: Using SegFormer for semantic segmentation provides a richer contextual understanding than traditional visual feature extractors, helping to avoid description errors and improve completeness.
- Cross-Attention for Enhanced Interaction: The cross-attention mechanism in the decoder is vital for effectively integrating the fused grid and segmentation features, leading to more coherent and accurate captions.
- Dataset Creation is a Significant Contribution: The development of dedicated maritime datasets (MSS and MIC) is a valuable contribution, enabling further research and development in this specialized domain.
Key Results
The TDSI model demonstrated superior performance compared to several baseline models, including standard Transformers, Meshed-memory Transformers, and Deep Fusion Transformers.
- Quantitative Improvements: TDSI achieved the highest scores across multiple evaluation metrics, including BLEU-4 (0.788), METEOR (0.575), ROUGE (0.897), and CIDEr (3.343) on the MIC dataset. Notably, the CIDEr score was 0.081 higher than a strong Transformer baseline and 0.029 higher than the DFT model.
- Qualitative Improvements: The qualitative analysis showed that TDSI generated more accurate and descriptive captions. For instance, in one example, TDSI correctly identified "city" while a baseline model incorrectly identified "islands." In another case, TDSI accurately described "buoys," whereas the baseline generated an irrelevant description. The paper also notes that TDSI's descriptions are more accurate and complete, effectively integrating segmentation and grid features.
Limitations and Future Work
- Dataset Size and Generalizability: The authors acknowledge that while they created new datasets, their size and generalizability might still pose challenges for training highly robust models. Future work should focus on expanding these datasets with more diverse scenes, regions, and varying conditions (time of day, weather).
- Object Percentage Influence: The accuracy of TDSI's descriptions can be affected when the pixel percentage of certain objects in the image is very low.
- Future Directions: The paper suggests expanding the dataset and exploring further advancements in model architecture and training strategies to improve adaptability to a wider range of real-world maritime scenarios and facilitate practical deployment.
Summary updated