CropCap: Embedding Visual Cross-Partition Dependency for Image Captioning
Lists
Only you see this
Abstract
Transformer-based approaches to image captioning have shown great success by utilizing long-term dependency for visual embedding. However, their coarse long-term dependency, using the multi-head self-attention mechanism to capture the contextual interactions between the visual tokens on the time step and (or) embedded dimension, fail to distinguish fine-grained features of local partition. In this case, some similar features are captured, which leads to feature redundancy that decreases the performance. To respond to this issue, this paper proposes a novel image captioner embedding visual cross-partition dependency, dubbed CropCap. Specifically, the visual sequence generated from the Swin Transformer-based pre-embedding network is fed into the proposed cross-partition dependency module to refinedly model the interaction between partial representations on both the time step and embedded dimension. Furthermore, we formulaically reason the proposed cross-partition dependency, and theoretically prove its correctness. Extensive comparisons on the benchmark MS-COCO dataset demonstrated the effectiveness addressing the information redundancy issue, and verified the superior performance of our method.
BibTeX
@inproceedings{wang2023cropcap,
author = {Bo Wang and Zhao Zhang and Suiyi Zhao and Haijun Zhang and Richang Hong and Meng Wang},
title = {{CropCap: Embedding Visual Cross-Partition Dependency for Image Captioning}},
booktitle = {ACM Multimedia},
year = {2023},
doi = {10.1145/3581783.3612245},
}Loading related content...