Archiving Body Movements: Collective Generation of Chinese Calligraphy

Aven Le Zhou, Jiayi Ye, Tianchen Liu, Kang ZhangarXiv 2023 · DOI · arXiv

Introduction

In the traditional practice of Chinese calligraphy, the physical performance—the "movements in the air"—is considered as vital as the final ink on the page. However, once a work is completed, this dynamic energy is often lost to the viewer, leaving behind only a static image. This paper introduces Wushu, an interactive AI-driven installation designed to bridge this gap by transforming human body movements into calligraphic art in real-time. By doing so, it creates a "collective digital archive"—an infinite book where the ephemeral energy of the audience is preserved through calligraphic characters and ink-wash aesthetics.

The Problem: Capturing Ephemeral Motion

While fields like Kinesics and performing arts utilize technical notation systems (such as Labanotation) to record movement, these systems often lack the expressive, painterly quality found in visual arts. Conversely, most contemporary AI art focuses on the final visual output rather than the generative process. The authors address a specific challenge: how to capture, represent, and archive the transient energy of human movement through the culturally rich and visually evocative lens of calligraphy.

The Wushu System Architecture

The authors developed a multi-stage pipeline to convert a participant's physical presence into a digital "infinite book":

  1. Body Pose Estimation: Utilizing Google MediaPipe, the system tracks skeletal joints, specifically the hands, in real-time to capture motion trajectories.
  2. Ink & Stroke Simulation: To provide immediate visual feedback, the system employs a Partial Differential Equation (PDE) based on optical flow algorithms. This simulates the fluid dynamics of ink spreading and absorbing into paper, effectively turning the participant's hand into a virtual brush.
  3. Motion-to-Text (Motion Captioning): This core component uses pre-trained Vision-Language models—BLIP for English and ViT-GPT2 for Chinese—to generate textual descriptions of the captured movement images. The AI essentially "interprets" the gesture into descriptive words.
  4. Generative Calligraphy & Layout: A custom algorithm renders these generated words using calligraphic fonts. The aspect ratio and layout of the characters are modulated by the original trajectory and "energy" of the participant's movement, adhering to traditional principles of character composition.

The Co-Creation Framework

The system is built around a dual-loop feedback mechanism that redefines the relationship between the viewer and the artwork.

The Wushu system framework illustrating the alpha and beta loops of the co-creation process.

As shown in the framework above, the α loop represents the immediate interaction where the visual beauty of the generated ink encourages more dynamic movement from the participant. The β loop represents the long-term archival process, where the collective contributions of many participants are stored in the "Archive." This shifts the audience from passive viewers to "participant-creators," where the final "book" is a collective effort accumulated over time.

Deployment and Results

The Wushu system was successfully deployed at major international venues, including the Ars Electronica Festival 2024 and CVPR 2024.

The Wushu installation in a gallery setting, showing participants interacting with the dual-screen display.

Observations during these exhibitions revealed that participants naturally adopted martial-arts-like stances to interact with the work, effectively "performing" the calligraphy. The system's bilingual interpretation—generating both English and Chinese captions—served as a cross-cultural translator, making the cultural essence of calligraphy accessible to non-Chinese speakers.

However, the authors noted a performance disparity between the English and Chinese captioning models. The English model (BLIP) produced more accurate and fluent descriptions, likely due to the higher quality and diversity of English-language training data available in the AI field. This highlights a significant area for future research regarding data bias in generative AI.

Limitations and Future Work

Currently, the "infinite book" is saved as a series of image files. Future iterations aim to explore more immersive archival formats, such as real-time online galleries or physical, printed compendiums that mimic traditional scrolls. Additionally, the team is working on refining interaction triggers to better distinguish intentional artistic movements from incidental ones, ensuring a more seamless creative experience.

Summary updated

BibTeX
@misc{zhou2023archiving,
  author = {Aven Le Zhou and Jiayi Ye and Tianchen Liu and Kang Zhang},
  title = {{Archiving Body Movements: Collective Generation of Chinese Calligraphy}},
  year = {2023},
  eprint = {2311.13770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.13770},
}
Figure