Text Location-Aware Framework for Chinese Meme Generation

Wang Lin, Ruizheng Wu, Pengchu Luo, Chang-Hun KimIEEE Access 2025 · DOI

Abstract

Automated meme generation has traditionally focused on textual content, often neglecting the crucial role of spatial arrangement in comedic impact. This paper presents a novel framework for Chinese meme generation that dynamically predicts optimal text placement. By combining multimodal models (ChineseCLIP, GPT-2) with an anchor-free object detection head (FCOS), the system ensures that generated text aligns with visual cues. Experimental results demonstrate that this approach significantly outperforms existing baselines, producing memes that are frequently indistinguishable from human-created content.

Introduction: The Importance of Placement

Internet memes are a sophisticated form of multimodal communication where the spatial relationship between text and image is vital. Most existing AI meme generators suffer from "rigid positioning," typically placing text at fixed top or bottom locations regardless of image content.

Example of a Chinese meme where text placement and visual cues interact.

In the world of memes, placement is everything. If a character is looking toward a specific corner, the punchline belongs there. The authors identify a major gap in Chinese meme generation: a lack of dynamic adjustment capabilities that allow the AI to "understand" the layout of the image and place text harmoniously.

Methodology

The researchers proposed a two-stage framework to handle both content and layout:

  1. Text Generation (The "What"): The system utilizes CLIPCap, bridging visual and textual spaces. It employs ChineseCLIP as the image encoder and GPT-2 as the text decoder. A multi-layer perceptron mapping network translates image features into "prefix" prompts to guide the generation of meme-style Chinese text.
  2. Text Positioning (The "Where"): This core innovation adapts FCOS (Fully Convolutional One-Stage object detection), an anchor-free model, to predict the ideal "text box."
  3. Fusion Attention: To ensure the location matches the message, a cross-modal attention mechanism combines image features (from ResNet) with text features (from Chinese RoBERTa). This allows the model to "attend" to specific image regions based on the semantic content of the generated text.
  4. Post-processing: An automatic text-drawing algorithm dynamically wraps text and adjusts font size to fit the predicted FCOS bounding box.

Dataset Construction

Due to a lack of existing datasets with text coordinates for Chinese memes, the authors built the Meme-Location Dataset. It consists of 3,000 Chinese memes collected from platforms like WeChat and Weibo. OCR was used to extract text and coordinates, followed by manual verification. This was supplemented with 30,000 image-caption pairs for robust training.

Key Insights and Results

A central insight of this work is treating the ideal location for a punchline as a detectable "object." This allows the model to leverage computer vision progress for graphic design. Furthermore, the meaning of the text dictates the placement.

A meme illustrating semantic-driven placement: the question "Where is your brain?" is placed near the character's head.

By using RoBERTa to encode text before feeding it into the FCOS head, the model learns that certain phrases belong near specific visual features, such as faces or empty negative space.

Performance Metrics: - Quantitative: The model achieved a BLEU-4 score of 0.500, a significant improvement over the baseline (0.098). It also reached a BERTScore of 0.580. - Human Evaluation: In a study of 50 frequent meme users, 70% preferred the dynamic text placement of this model. Notably, 64.4% of the AI-generated memes were mistaken for real, human-made memes.

Conclusion and Future Work

While the framework is a major leap forward, limitations remain in handling exceptionally long text and avoiding features that trigger "AI-detection" in humans. Future work will explore broader applications, such as automated advertisement design and social media content creation, where location-aware multimodal generation is equally critical.

Summary updated

BibTeX
@article{lin2025text,
  author = {Wang Lin and Ruizheng Wu and Pengchu Luo and Chang-Hun Kim},
  title = {{Text Location-Aware Framework for Chinese Meme Generation}},
  journal = {IEEE Access},
  year = {2025},
  doi = {10.1109/access.2024.3401174},
}
Figure