Haiku Generator that Reads Blogs and Illustrates Them with Sounds and Images

Rafal Rzepka, Kenji ArakiIJCAI 2015

Introduction

Artificial art often struggles with the "uncanny valley" of creativity, frequently appearing sterile or nonsensical due to a lack of context. Traditional poetry generators have historically been limited to static vocabularies or produced abstract results that failed to resonate with a reader's immediate situation. To address this, researchers presented a system at IJCAI 2015 that bridges digital life and traditional art by generating haikus based on user blog posts.

The system aims to inhabit the "Internet of Things" by appearing on screens or devices in response to daily activities. For haiku—a genre defined by its connection to the "now"—the generator must capture the specific mood and season of the user's environment. By using web-mining to ensure semantic integrity and "emotional unison," the system produces contextually relevant poems supplemented by synthesized speech, sound effects, and imagery.

System Architecture

The generation process follows a multi-stage pipeline designed to transform a raw blog entry into a cohesive multimedia experience.

The multi-stage pipeline for haiku generation from blog content.
  1. Topic Discovery: The system analyzes a blog post using the MeCab morphological analyzer to identify nouns. A "Theme Score," based on word frequency and web-mined importance, is used to select a central theme (e.g., "Mt. Moiwa").
  2. Semantic Integrity via Web-Mining: To ensure coherence, the system leverages "the wisdom of the web-crowds." It queries search engines to calculate co-occurrence probabilities between the theme, potential adjectives, and the blog's title. This ensures that if a blog discusses a "beautiful view," the haiku utilizes adjectives commonly associated with that location by humans.
  3. Kigo (Seasonal Word) Selection: Haikus require a kigo. The system selects an appropriate term from a database of 2,473 seasonal words by identifying those that co-occur most frequently with the chosen theme on the web.
  4. Template-Based Generation: Using seven grammatical patterns extracted from a corpus of 20,000 human-written haikus, the system populates templates with the mined nouns, adjectives, and kigo.
  5. Multimedia Enrichment: The final poem is read aloud via a TTS engine, accompanied by related images and sound effects associated with the poem's nouns.

Evaluation and Results

The system was evaluated using Semantic Differential (SD) testing and direct comparisons against a baseline random-word generator. Human evaluators found the proposed system significantly more "deep" and "sharp" than the baseline.

Comparative evaluation of the proposed system against a baseline generator.

As shown in the results, the proposed system outperformed the baseline across all metrics, including "Seasonal Impression," "Haiku Quality," and "Willingness to Use." Notably, the system achieved a high score in "Sound Rhythm" and "Scene Imagination," suggesting that the combination of web-mined semantics and multimedia output effectively conveys poetic intent.

Interestingly, the researchers found that multimedia elements can act as a "buffer," masking minor linguistic awkwardness. For instance, an image of a camisole (representing summer) helps readers bridge the gap between text and emotion. However, evaluation also revealed that blog authors were harsher critics than third-party readers, likely because authors have specific emotional intents that the AI may not fully capture.

Conclusion and Future Work

The research demonstrates that tying AI generation to immediate human experiences—like recent blog posts—can create more "natural" and resonant artificial art. While the current system relies on static templates and sometimes misses specific authorial nuances, it provides a robust framework for context-aware creativity. Future work will focus on developing interactive modules for human-machine collaboration and refining the multimedia mix to find the optimal balance of sensory inputs for various poetic moods.

Summary updated

BibTeX
@inproceedings{rzepka2015haiku,
  author = {Rafal Rzepka and Kenji Araki},
  title = {{Haiku Generator that Reads Blogs and Illustrates Them with Sounds and Images}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure