Application of AR virtual implantation technology based on deep learning and emotional technology in the creation of interactive picture books

Sidan Liu, Peng Peng, Lei Cao3C 2023 · DOI

Abstract

In recent years, deep learning has transitioned from solving traditional algorithmic problems to enhancing product emotional design. This paper explores the integration of deep learning and affective technology in Augmented Reality (AR) interactive picture books. By transforming the reading process from static to dynamic, the proposed framework aims to enrich visual stimulation and increase interactivity. Based on a three-level emotional design model, image emotion labeling results were used to train a deep neural network. The resulting model demonstrates a significant improvement in emotional recognition accuracy compared to traditional algorithms, addressing the "semantic gap" in current AR marketing and meeting children's emotional reading needs.

Introduction

As digital media increasingly competes for children's attention, traditional paper-based picture books face challenges in engagement. While Augmented Reality (AR) offers a potential solution, many current implementations are dismissed as "marketing gimmicks"—3D models that lack a meaningful connection to the narrative's emotional core. The primary obstacle is the semantic gap: the difficulty for computational systems to interpret high-level emotional concepts like "trust" or "comfort" from low-level visual data like pixels and shapes. This research proposes bridging this gap by combining deep learning architectures with established emotional design theories.

Theoretical Framework: Emotional Design

The foundation of this study rests on Donald Norman’s three-level theoretical model of emotional design. This model categorizes user experience into three distinct layers:

  1. Instinctive Design: Focuses on immediate sensory impact, such as color, texture, and form.
  2. Behavioral Design: Relates to the functionality, usability, and the joy of interaction.
  3. Reflective Design: Concerns deep consciousness, cultural meaning, and long-term psychological impact.
The three-level theoretical model of emotional design: Instinctive, Behavioral, and Reflective.

By applying this framework, designers can move beyond superficial visual effects to create AR content that resonates with a child's psychological development.

Methodology and Emotional Classification

To automate the emotional analysis of picture book illustrations, the researchers selected the VGGNet deep learning architecture. VGGNet was chosen for its superior generalization capabilities in computer vision. Its use of small (3x3) convolutional kernels allows the network to capture nuanced "sensory fields" essential for identifying subtle emotional cues.

Dataset and Labeling

A custom "Preliminary Design Gallery" was constructed, consisting of 2,048 images sourced from award-winning product designs. These images were selected for their clear emotional characteristics and labeled by design professionals across 11 categories: Easy, Comfort, Simple, Trust, Softness, Happy, Noble, Surprise, Ordered, Lovely, and Belonging.

Plutchik's Wheel of Emotions, illustrating the spectrum of human sentiment used to guide image labeling.

The dataset was split into 1,500 training images and 548 testing images. The labeling process revealed that "Easy" (22.7%) and "Comfort" (12.15%) were the most prevalent emotions, suggesting a current industry focus on the "Behavioral" level of design.

Results and Discussion

The VGGNet model achieved a single-label prediction accuracy of 61.02%. This represents a notable advancement over traditional machine learning algorithms, which typically range between 50% and 60% for similar emotional classification tasks.

Key findings include:

  • Deep Learning Efficacy: While human observers often find it difficult to articulate why an image feels "noble" or "soft," the deep neural network successfully "learned" these patterns through backpropagation and weight optimization.
  • The Reflective Gap: The "Reflective" layer (e.g., "Belonging") was the least represented in the dataset (1.71%). This highlights a significant opportunity for future interactive media to deepen its emotional and cultural resonance.

Conclusion

The integration of AR virtual implantation with deep learning-based emotional recognition offers a path toward more meaningful interactive picture books. By moving beyond simple 3D overlays and aligning digital content with the emotional arc of a story, creators can better stimulate children's creativity and engagement. Future research will focus on expanding the image dataset and specifically targeting the "Reflective" layer of emotional design to foster long-term psychological benefits.

Summary updated

Abstract

In recent years, the field of deep learning has flourished, not only breaking through many difficult problems that are difficult to be solved by traditional algorithms but also bursting with greater vitality when combined with other fields. For example, product emotional design based on deep learning can integrate users' emotional needs into the actual product design. In this paper, we aim to use deep learning and affective technology in the creation of AR interactive picture books to transform the reading process from static to dynamic, enrich visual stimulation, and increase the fun and interactivity of reading. In this paper, based on the three-level theoretical model of emotion, the emotion labeling results are input to a deep neural network for learning, to establish an emotion-based recognition model for picture book images. The results show that the model can well analyze the emotion of images in AR picture books, and the accuracy of prediction is a big improvement compared with traditional machine recognition algorithms. The application of AR virtual implantation technology in interactive picture books on the market is often just a marketing gimmick while combining deep learning and emotional technology can better create diverse interactive picture books to meet children's emotional reading needs, enhance reading engagement, and stimulate children's creativity.
BibTeX
@inproceedings{liu2023application,
  author = {Sidan Liu and Peng Peng and Lei Cao},
  title = {{Application of AR virtual implantation technology based on deep learning and emotional technology in the creation of interactive picture books}},
  booktitle = {3C},
  year = {2023},
  doi = {10.17993/3ctic.2023.121.176-198},
}
Figure