Designers of expressive interfaces, such as animated chat bubbles, stylized robot faces, and android heads, judge their work by one number. They show each design once to naive viewers and count how often the viewers name the intended emotion. That single-session score now serves as the main evidence that a design "works." It plays the same role that short-term biomarkers play in clinical trials: cheap to measure, assumed to predict the outcome that matters, and almost never checked against it. We propose to check it. We will deploy a set of animated chat balloons in real messaging between pairs of people for twelve weeks, with a delayed probe at six months. The key design choice is to release different balloon designs to different pairs at randomized, staggered times. Because a given week contains some designs that are new to a pair and others that are familiar, we can separate three things a short study cannot tell apart: a general novelty boost that fades with calendar time, habituation to a specific animation as it is seen more often, and the opposite effect, where pairs learn what an ambiguous animation means.
The core question is whether the lab recognizability score ranks designs in the same order as their long-term value. Here long-term value means whether receivers still read the sender's intended feeling correctly in context, and whether senders keep choosing the design. If the ranking holds, the field gains justified confidence in a cheap metric. If it fails, the failure takes one of two shapes. Highly legible designs may wear out, or low-legibility designs may become meaningful through use. Either shape changes how expressive interfaces should be evaluated.
Core Claim
The claim is that single-session recognizability is an unvalidated surrogate, and that its validity can be measured directly. The test asks how well per-design lab accuracy predicts per-design in-context interpretation agreement and continued use after weeks of exposure. A field study whose exposure is randomized in time can split per-design change into exposure-driven and calendar-driven parts. We also test a practical hypothesis that, if true, would make long field studies unnecessary for this purpose: a compressed lab session with many repetitions of the same design may predict field habituation slopes design by design. That hypothesis is a skeptic's preferred shortcut, and we treat it as something to measure.
Grounding in Prior Work
The stimuli come from AniBalloons and its early exploration. That work built 30 kinetic chat-balloon designs covering six emotions: joy, anger, fear, surprise, sadness, and calmness. Shown without any text, 24 of the 30 designs exceeded 50% recognition and 20 exceeded 65%. In single-session scripted chats and chatbot conversations, the balloons raised perceived expressiveness, closeness, and chatbot likability. The authors state that their studies used mockups and that longitudinal, real-world deployment is needed. We take their design set and their recognition protocol unchanged, and supply the deployment they call for.
Two other works establish that recognizability is not fixed. The robot Nikola was evaluated on Facial Action Coding System units and still and moving stimuli. Its happy expressions were recognized 98.2% of the time, with negative emotions recognized less well. The abstract-face study of Wang et al. placed stylized faces in a three-axis pleasure-arousal-dominance space and tuned eyebrow geometry to reduce the distance between each face and its target emotion. Both treat recognizability as a property of the design measured at one moment. Xu et al. cut against that assumption. Brief exposure to a simple curved line shifts later judgments of facial expression. The effect is so tied to retinal position that a 0.2° shift in where the line appears removes it. This motivates the question but does not answer it. It shows that perception of simple affective shapes adapts on a scale of seconds. It does not show whether anything like that operates across weeks of everyday use, where position on the retina varies constantly and slower cognitive habituation or learning would dominate instead.
The second direction, limited follow-up duration, supplies the methodological lens.
Clinical trials repeatedly find that short windows hide decay. In ADL+, gains measured at the end of the 6-month intervention looked different, domain by domain, at the 9-month follow-up. The VR trials of Kwan et al. and Wong et al. report large immediate gains with almost no later follow-up. In Wong et al., psychologist-rated social skills improved with d = 1.50 against traditional training, with no follow-up at all. Kwan et al. lost 14.4% of the intervention arm within eight weeks, which sets our expectation for dropout.
Outside medicine, Zheng et al. show that the spending lift from installing an EV charger fell from 1.4% in 2019 to 0.8% in 2021–2023. A short window would have overstated the lasting effect.
The closest longitudinal human-robot study, Stock-Whittaker et al., followed children over three weekly sessions with a robot. Their positive perceptions did not decline. That result partly contradicts the novelty-decay story. We take it seriously: three weeks may simply be too short, and the authors say months are needed. It also involved no manipulation of affective expression.
Zahedi et al. model how trust changes over repeated rounds and argue that the best robot behavior shifts as familiarity grows. That supports our premise that the value of an expressive signal depends on the history of the relationship. Their "longitudinal" setting, however, was a single session lasting about an hour.
Research Program
We will build a lightweight messaging app for pairs of people who already know each other. Senders can attach one of the 30 AniBalloons animations to any message. A matched control arm offers static stickers covering the same six emotions instead. The 30 designs are split into three release waves of ten, balanced so that each wave covers the six emotion categories. For each pair, a random assignment decides which wave unlocks in week 0, week 4, and week 8. Every design therefore appears to some pairs at every stage of their app history, and within any one pair, old and new designs coexist.
When a sender attaches a balloon, the app occasionally and privately asks which feeling the sender intended. The receiver is occasionally asked what feeling they perceived. Every two weeks, each participant also completes a five-minute probe that repeats the AniBalloons protocol exactly. They see designs without any text, choose one of six emotions, and rate valence and arousal on 7-point scales. The probe includes designs the participant has seen many times and designs they have never seen. Participants also give weekly ratings of expressiveness and annoyance, plus the interpersonal-closeness measures used by An et al.
For each participant $i$, design $d$, and probe occasion, we model the probability of correct recognition $y$ as
$$\text{logit}\, P(y_{idt}=1) = \alpha_d + \beta\, \tau_{it} + \gamma_d\, \log(1+n_{idt}) + u_i + v_{\text{pair}},$$
The terms are defined as follows. $\alpha_d$ is the design's baseline legibility. $\tau_{it}$ is the number of weeks since participant $i$ joined the app, which captures general novelty. $n_{idt}$ is how many times that participant has sent or received design $d$ so far. $\gamma_d$ is the design-specific exposure slope: negative means habituation, positive means learning. $u_i$ and $v_{\text{pair}}$ are random effects for person and pair.
We use a logarithm of exposure count because habituation and learning curves typically flatten after early repetitions. Model comparison will check this choice against alternative curve shapes.
The staggered release is what makes $\beta$ and $\gamma_d$ separately identifiable. If every design were released on day 0, exposure count and calendar time would rise together and the two could not be told apart. The same model, with in-context sender–receiver agreement as the outcome and with continued sender choice of design $d$ as the outcome, gives the two measures of long-term value.
Alongside the field study, we run a compressed-exposure lab session with a separate sample. Each design is shown about forty times over an hour, interleaved with other designs, and recognized after each block. This gives a lab estimate of each design's exposure slope, which we compare against the field estimates of $\gamma_d$.
Baselines
The naive reference point is the assumption the field currently makes: recognizability does not change. Under this baseline, each design's single-session accuracy from our own week-0 probe predicts its in-context agreement and usage at week 12 and month 6, and nothing else does. To beat it, a model that adds exposure and calendar terms must explain significantly more held-out variance across designs and participants under cross-validation by pair. The decisive check is whether the rank correlation between baseline accuracy and week-12 agreement falls clearly short of the split-half reliability of the week-12 measure itself. A shortfall means the surrogate is losing information.
The strongest existing method is the AniBalloons evaluation protocol from An et al., run exactly as published. Its reported result is that 80% of designs (24 of 30) exceeded 50% decontextualized recognition and 20 exceeded 65%. We will replicate this at week 0 on our own sample, with the same designs, response options, and scales. Their reported likability and expressiveness gains in chat are significance results; no numeric effect size for them was found on Lacuna. Our version of their claim is the gain over the static-sticker arm at week 0. Our test is whether that gain survives at week 12 and at month 6.
The static-sticker arm is the "just use what people already have" baseline. It receives the identical staggered release and probe schedule. That allows it to separate decay specific to animated designs from decay of any affect add-on.
The compressed-exposure lab protocol is the cheap competitor. It beats our field estimate in practical terms if its per-design exposure slopes correlate with the field slopes about as strongly as the field slopes correlate across two random halves of the field sample.
The key ablation removes the stagger. It is a smaller arm in which all 30 designs unlock on day 0. In this arm, $\beta$ and $\gamma_d$ should become confounded, which shows up as inflated uncertainty and unstable estimates. This demonstrates that the release design, not the sample size, is what makes the decomposition possible.
Why not just…?
Why not just compare novice and veteran users cross-sectionally? For newly created designs there are no veterans. People who adopt an expressive feature early and keep using it are also self-selected toward liking it, so any difference between groups mixes selection with exposure. The ADL+ follow-up shows why this matters: different outcomes changed differently between 6 and 9 months, and a single snapshot would have hidden those patterns. Our within-pair stagger yields the novice-versus-veteran contrast inside the same people.
Why not just run many repetitions in one lab session? This is the real threat, and we test it directly instead of arguing against it. If the compressed session recovers each design's field slope, that is a positive finding: future designers can screen for wear-out in an hour. We expect the shortcut to miss learning effects. In those cases meaning comes from repeated use in conversation with a particular partner, which a lab session cannot provide. We also expect it to overstate short-lived adaptation of the kind Xu et al. found, which disappears with small changes in viewing position. Designs whose field slope is positive while their lab slope is negative would settle that question.
Why not just use static emojis, whose meanings are already settled? That may turn out to be right in the long run, which is why static stickers form a full arm rather than a footnote. An et al. found animated balloons more expressive than plain text in a single session. If their advantage over stickers disappears by week 12, the honest conclusion is that animation's benefit was novelty, and we will report it that way.
Why not just log usage and skip the probes? Usage reflects the relationship, the mood, and the topic of conversation. It cannot reveal whether the receiver decoded the intended feeling. The paired sender-intent and receiver-perception reports measure whether communication succeeded, which usage logs cannot.
Proposed Experiments
The field study enrolls 140 pairs, budgeting for roughly 15% dropout in line with Kwan et al.. One hundred pairs go to the staggered AniBalloons arm, twenty to the staggered sticker arm, and twenty to the unstaggered ablation. Active use runs twelve weeks, followed by a delayed probe at six months. We will model dropout explicitly and report analyses under assumptions that dropout is related to the outcome. The compressed-exposure lab study recruits a separate sample of 60 participants.
Four results would be decisive.
The first is the size of the gap between the baseline-accuracy ranking of designs and their week-12 in-context agreement and sender retention. This determines whether the surrogate holds.
The second is the sign and spread of the design-specific exposure slopes $\gamma_d$ after accounting for calendar novelty $\beta$. In particular: do the low-legibility sadness and calmness designs learn, with positive slopes, while the joy and surprise designs habituate, with negative slopes? That pattern would mean the valence asymmetry seen in Nikola and in AniBalloons shrinks with use instead of widening.
The third is whether the AniBalloons-over-sticker gain in expressiveness and closeness persists beyond the three-week horizon at which Stock-Whittaker et al. found no decline.
The fourth is how well the compressed lab session predicts field slopes, design by design.
Together, these results tell the field whether one-shot recognizability scores can keep serving as evidence that an expressive design works, and if they cannot, which cheaper check could replace them.
References
- AniBalloons: Animated chat balloons as affective augmentation for social messaging and chatbot interaction
- Affective Affordance of Message Balloon Animations: An Early Exploration of AniBalloons
- An Android for Emotional Interaction: Spatiotemporal Validation of Its Facial Expressions
- Experimental Study on Abstract Expression of Human-Robot Emotional Communication
- Adaptation across the Cortical Hierarchy: Low-Level Curve Adaptation Affects High-Level Facial-Expression Judgments
- ADL+: A Digital Toolkit for Multidomain Cognitive, Physical, and Nutritional Interventions to Prevent Cognitive Decline in Community-Dwelling Older Adults
- Effects of Virtual Reality Motor-Cognitive Training for Older People With Cognitive Frailty: Multicentered Randomized Controlled Trial
- Empowering Social Growth Through Virtual Reality–Based Intervention for Children With Attention-Deficit/Hyperactivity Disorder: 3-Arm Randomized Controlled Trial
- Effects of electric vehicle charging stations on the economic vitality of local businesses
- A Longitudinal Study of Child Wellbeing Assessment via Online Interactions with a Social Robot
- Trust-Aware Planning:Modeling Trust Evolution in Longitudinal Human-Robot Interaction