Query-based Image Captioning from Multi-context 360cdegree Images

Koki Maeda, Shuhei Kurita, Taiki Miyanishi, Naoaki OkazakiEMNLP 2023 · OpenReview

Abstract

360-degree images capture entire environments, presenting a significant challenge for traditional image captioning models that typically focus on a single salient object. This paper introduces Query-based Image Captioning (QuIC), a novel task where users guide caption generation using short text queries to describe specific sub-contexts within an immersive scene. We present the QuIC-360° dataset, containing over 18,000 query-caption pairs for panoramic images. Our experiments with fine-tuned vision-language models demonstrate that QuIC significantly improves caption relevance and diversity, successfully overcoming the "salience trap" of standard models.

Introduction: The Challenge of 360-degree Scenes

Traditional image captioning models are optimized for standard, narrow field-of-view photos where the primary subject is usually obvious. However, 360-degree (omnidirectional) images—increasingly vital in autonomous driving, robotics, and VR—capture the entire visible surroundings indiscriminately.

A 360-degree panoramic image illustrating the complexity and multiple contexts of an immersive scene.

A single caption for such an image often fails because the scene contains multiple, equally important contexts. Existing models tend to default to the most prominent feature, a phenomenon the authors call the "salience trap," while ignoring minor but relevant details. To truly understand these rich environments, models require a mechanism to focus on specific aspects of the scene based on user interest.

The QuIC-360° Paradigm

To address this, the authors propose the QuIC-360° task: given a 360-degree image and a textual query (a word or short phrase), the model must generate a relevant caption. This differs from standard captioning and controllable captioning (which uses structural hints like POS tags) by using semantic context as the control signal.

Comparison of Standard, Controllable, and the proposed Query-based Image Captioning (QuIC) paradigms.

Dataset and Methodology

The researchers introduced the QuIC-360° dataset, the first of its kind, featuring 3,940 panoramic images and 18,459 pairs of queries and manually written captions. They defined 34 scene queries (e.g., Location, People, Weather, Building, Traffic) based on common elements in 360-degree scenery. Captions in this dataset average ~19.4 words, reflecting the complexity of the scenes compared to standard datasets like MS-COCO.

To solve the QuIC task, the authors adapted two state-of-the-art vision-language models: OFA (One-For-All) and BLIP-2. The key technical modification involved prompt engineering; instead of generic instructions, the models were prompted with queries like "What does this image describe on [query]?" to force attention toward specific visual features.

Results and Key Insights

Fine-tuning on the QuIC-360° dataset yielded dramatic improvements. For the OFA-base model, the CIDEr score jumped from 8.5 to 26.9 when the query was introduced. Human evaluations confirmed this, with query relevance scores improving from 3.40 to 3.70 on a 5-point scale.

One critical finding was that CLIP-based metrics (like CLIPScore) are unsuitable for this task. While human evaluators saw improvement, CLIPScore remained flat. This is because CLIP evaluates global similarity between a caption and the entire image, whereas QuIC requires focusing on local details that may occupy only a small portion of the 360-degree view.

Conclusion

QuIC-360° provides a path toward more controllable and context-aware AI for immersive environments. While models still occasionally "hallucinate" objects to satisfy a query (e.g., inventing a train in a forest), the ability to generate multiple, distinct, and accurate captions for a single image marks a significant step forward. Future work will focus on expanding the dataset scale and explicitly grounding queries to spatial coordinates within the 360-degree projection.

Summary updated

BibTeX
@inproceedings{maeda2023query,
  author = {Koki Maeda and Shuhei Kurita and Taiki Miyanishi and Naoaki Okazaki},
  title = {{Query-based Image Captioning from Multi-context 360cdegree Images}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=20kSv6aAB7},
}
Figure