Captioning Bosch: A Twitter Bot

Cornelia FernerIJCAI 2022 · DOI

TL;DR

This paper introduces CaptioningBoschBot, a Twitter bot that uses a pre-trained image captioning model to interpret segments of Hieronymus Bosch's intricate paintings. By applying a model trained on realistic images to surreal art, the bot generates often absurd but thought-provoking captions, sparking discussions about AI's interpretation of art and its limitations.

What Problem Does This Paper Address?

Hieronymus Bosch's artworks are renowned for their dense detail and surreal imagery, making them fascinating but also challenging to fully appreciate. While existing art bots like BoschBot present these details to users, this paper tackles the inverse problem: how can an AI interpret and describe these complex, non-realistic visual scenes? The authors aim to explore the capabilities and limitations of current image captioning models when applied to art that falls outside their typical training domain of photorealistic images. This exploration serves to highlight the gap between AI's objective, data-driven understanding and human artistic interpretation, which is informed by cultural context and historical understanding.

Methods

The CaptioningBoschBot operates in three main steps:

  1. Accessing Tweets: The bot uses the Twitter API to retrieve the latest tweets from the existing BoschBot. It then extracts the image URL from these tweets.
  2. Generating Captions: A pre-trained vision-to-text transformer model is employed for image captioning. This model consists of a Vision Transformer (ViT) as the encoder and a GPT-2 neural network as the decoder. The ViT is pre-trained on ImageNet, and the entire model is pre-trained on the MS COCO captions dataset. The model generates captions using a beam search with 4 beams and a maximum sequence length of 16 tokens.
  3. Posting Retweets: The generated caption is then used to create a retweet of the original BoschBot post, thereby sharing the AI's interpretation with the Twitter audience.

The novelty lies in applying a standard image captioning pipeline, trained on realistic data, to a domain (surreal historical art) it was not designed for.

Datasets

  • MS COCO Captions Dataset: Used for pre-training the encoder-decoder image captioning model. This dataset contains over 330,000 images with corresponding captions, primarily featuring real-world scenes.
  • ImageNet: Used for pre-training the Vision Transformer (ViT) encoder.

The paper does not introduce new datasets but rather highlights the mismatch between the training data (MS COCO) and the target domain (Bosch paintings).

Key Insights and Ideas

  • AI's "Literal" Interpretation of Art: When an AI trained on realistic images encounters surreal art, its interpretations can be surprisingly literal and often miss the intended artistic or historical context. This leads to captions that are factually descriptive of visible elements but lack deeper meaning.
  • The "Fixation on Cracking" Phenomenon: The model frequently identifies and describes the craquelure (cracking) of aged paintings, leading to repetitive captions like "a painting of a painting of a painting..." This highlights how visual artifacts not present in training data can become focal points for the model.
  • Loss of Focus on Complex Scenes: For densely populated or complex image segments, the model struggles to identify a single focal point, often resorting to generic descriptions like "a collage of photos of people and animals."
  • Misinterpretation of Historical Context: The AI's lack of cultural and historical understanding leads to anachronistic interpretations, such as identifying objects as "surfboards," "graffiti," or "frisbees" in historical artworks. It also exhibits a peculiar interpretation of nudity, often associating it with bathing or beach scenes.
  • Stimulating Human Imagination: While the AI's interpretations are often inaccurate, they can paradoxically spark human curiosity and encourage viewers to re-examine the artwork and consider why the AI might have generated such a caption.

Key Results

The paper presents qualitative results through examples of generated captions and their corresponding image segments.

  • Accurate Descriptions: In some cases, the model generates accurate descriptions of familiar objects within the Bosch paintings.
  • Absurd Interpretations: Numerous examples demonstrate the model's misinterpretations, such as describing a figure as "a painting of a woman sitting on a chair" when other elements are present, or identifying historical scenes with modern objects like "surfboards" or "frisbees."
  • Repetitive Captions: The "fixation on cracking" leads to captions that are often repetitive and overemphasize the painting's age.
  • Limited Vocabulary: The model's vocabulary, derived from realistic image datasets, struggles to describe the unique elements and surrealism present in Bosch's work.

Quantitative results are not explicitly detailed in terms of accuracy metrics, as the focus is on the qualitative exploration of the model's behavior on this specific artistic domain.

Limitations and Future Work

  • Training Data Bias: The primary limitation is the model's training on realistic, everyday scenes, which makes it ill-equipped to interpret artistic styles and historical contexts.
  • Inability to Grasp Artistic Intent: The AI lacks the capacity to understand artistic intent, symbolism, or the historical and cultural background of the artwork.
  • Focus on Central Objects: The model tends to focus on the most prominent object in the image center, often ignoring other details or background elements.

Future work suggested includes:

  • Omitting Repetitive Captions: Implementing filters to remove captions that exhibit repetitive patterns.
  • Training on Diverse Datasets: Training the encoder-decoder model on more diverse image datasets, potentially including art-related datasets, to improve its understanding of different visual styles.
  • Image Patch Captioning: Generating separate captions for different image patches to address the issue of lost focus in complex scenes.

Summary updated

BibTeX
@inproceedings{ferner2022captioning,
  author = {Cornelia Ferner},
  title = {{Captioning Bosch: A Twitter Bot}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/694},
}
Figure