Cycle-Consistency Learning for Captioning and Grounding

Ning Wang, Jiajun Deng, Mingbo Jia

Abstract: We present that visual grounding and image captioning, which perform as two mutually inverse processes, can be bridged together for collaborative training by careful designs. By consolidating this idea, we introduce CyCo, a cyclic-consistent learning framework to ameliorate the independent training pipelines of visual grounding and image captioning. The proposed framework (1) allows the semi-weakly supervised training of visual grounding; (2) improves the performance of fully supervised visual grounding; (3) yields a general captioning model that can describe arbitrary image regions. Extensive experiments show that our fully supervised grounding model achieves state-of-the-art performance, and the semi-weakly supervised one also exhibits competitive performance compared to the fully supervised counterparts. Our image captioning model has the capability to freely describe image regions and meanwhile shows impressive performance on prevalent captioning benchmarks.

Introduction

Visual grounding and image captioning are two fundamental tasks in vision-language research. Visual grounding aims to locate a specific region in an image based on a textual description, while image captioning generates a description for a given image or region. Despite their conceptual relationship, these tasks are traditionally treated as independent pipelines. This independence leads to two major bottlenecks: data scarcity, as visual grounding requires expensive "image-text-box" triplets, and granularity mismatch, as standard captioning models describe entire images rather than specific regions.

The CyCo Framework

To address these challenges, we introduce CyCo (Cycle-Consistency), a framework that bridges visual grounding and image captioning by treating them as mutually inverse processes. By establishing a closed-loop learning mechanism, the model can leverage "weak" data—images with only text or only bounding boxes—to improve performance on both tasks.

The framework operates through two primary cycles:

  1. Grounding-to-Captioning (VG → IC): The model takes a text prompt, predicts a bounding box, and then generates a new caption based on that box. The loss is computed by comparing the final caption to the original prompt. This allows the model to learn from images that have text labels but no bounding boxes.
  2. Captioning-to-Grounding (IC → VG): Starting with a bounding box, the model generates a regional caption and then attempts to ground that caption back to the original box. This allows the model to learn from images that have boxes but no specific text descriptions.
The CyCo framework utilizes dual cycles (VG → IC and IC → VG) to enforce consistency between grounding and regional captioning.

Architecture and Implementation

CyCo is built upon the BLIP architecture, utilizing a ViT-B/16 visual encoder and a Transformer-based text encoder/decoder. A key innovation is the partial weight-sharing between the grounding and captioning branches. Specifically, the cross-attention and feed-forward layers are shared, forcing the model to learn a unified representation for both tasks.

To enable regional awareness in the captioner, we incorporate a box-projection layer that allows the model to focus on specific image coordinates. This enables the model to describe arbitrary sub-regions with high precision, effectively acting as an online data augmentation engine through pseudo-labeling.

Detailed architecture of CyCo, highlighting the shared visual encoder and the specialized heads for grounding and regional captioning.

Experimental Results

The model was pre-trained on a large-scale collection of 115M images (including Visual Genome, MS-COCO, and LAION) and fine-tuned on standard benchmarks like RefCOCO.

  • State-of-the-Art Grounding: In fully supervised settings, CyCo achieved a top-1 accuracy of 89.47% on RefCOCO (val), outperforming previous strong models like VILLA and ERNIE-ViL.
  • Semi-Weakly Supervised Learning: CyCo demonstrates remarkable efficiency with limited data. Using only 20% of fully annotated triplets alongside 80% weak data, the model remains highly competitive, significantly closing the performance gap with fully supervised counterparts.
  • Enhanced Captioning: On the COCO benchmark, CyCo achieved a CIDEr score of 133.9, surpassing the baseline BLIP model. This suggests that learning to ground regions improves the overall quality and descriptiveness of the captioning model.

Conclusion

CyCo demonstrates that visual grounding and regional image captioning are synergistic tasks. By formalizing them as inverse processes within a cycle-consistency loop, we can overcome data scarcity and improve model granularity. While performance is currently tied to backbone scale and resolution, the success of this cyclic approach opens new avenues for other inverse vision-language tasks, such as image generation and retrieval.

Summary updated

Abstract

We present that visual grounding and image captioning, which perform as two mutually inverse processes, can be bridged together for collaborative training by careful designs. By consolidating this idea, we introduce CyCo, a cyclic-consistent learning framework to ameliorate the independent training pipelines of visual grounding and image captioning. The proposed framework (1) allows the semi-weakly supervised training of visual grounding; (2) improves the performance of fully supervised visual grounding; (3) yields a general captioning model that can describe arbitrary image regions. Extensive experiments show that our fully supervised grounding model achieves state-of-the-art performance, and the semi-weakly supervised one also exhibits competitive performance compared to the fully supervised counterparts. Our image captioning model has the capability to freely describe image regions and meanwhile shows impressive performance on prevalent captioning benchmarks.
BibTeX
@inproceedings{wang2024cycle,
  author = {Ning Wang and Jiajun Deng and Mingbo Jia},
  title = {{Cycle-Consistency Learning for Captioning and Grounding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=tohlznVWHa},
}
Figure