Statistical Inference for Regression with Imputed Binary Covariates with Application to Emotion Recognition

Ziqian Lin, Danyang Huang, Ziyu Xiong, Hansheng Wang

Abstract

In the flourishing live streaming industry, accurate recognition of streamers' emotions has become a critical research focus, with profound implications for audience engagement and content optimization. However, precise emotion coding typically requires manual annotation by trained experts, making it extremely expensive and time-consuming to obtain complete observational data for large-scale studies. Motivated by this challenge in streamer emotion recognition, we develop here a novel imputation method together with a principled statistical inference procedure for analyzing partially observed binary data. Specifically, we assume for each observation an auxiliary feature vector, which is sufficiently cheap to be fully collected for the whole sample. We next assume a small pilot sample with both the target binary covariates (i.e., the emotion status) and the auxiliary features fully observed, of which the size could be considerably smaller than that of the whole sample. Thereafter, a regression model can be constructed for the target binary covariates and the auxiliary features. This enables us to impute the missing binary features using the fully observed auxiliary features for the entire sample. We establish the associated asymptotic theory for principled statistical inference and present extensive simulation experiments, demonstrating the effectiveness and theoretical soundness of our proposed method. Furthermore, we validate our approach using a comprehensive dataset on emotion recognition in live streaming, demonstrating that our imputation method yields smaller standard errors and is more statistically efficient than using pilot data only. Our findings have significant implications for enhancing user experience and optimizing engagement on streaming platforms.

Introduction

In modern data science, understanding how human behavior or emotional states affect outcomes is a central goal. For instance, in the live streaming sector, researchers often ask: Does a streamer’s positive emotion lead to more user engagement? Answering this requires a regression model where emotion serves as a primary covariate. However, accurately determining emotion typically requires human experts to manually code video clips—a process that is logistically and financially impossible at the scale of millions of observations.

Researchers often face a dilemma: either use only the small subset of data they can afford to label (wasting the majority of the dataset) or use machine learning predictions as "ground truth" (introducing bias and underestimating uncertainty). This paper introduces a principled statistical framework to bridge this gap, allowing for valid inference using imputed binary covariates.

Methodology

The authors propose a two-stage approach that utilizes a small "pilot" of hand-labeled data and a large set of inexpensive auxiliary features (such as audio embeddings) to impute missing values and conduct valid statistical inference.

Overview of the proposed imputation and substantive modeling framework.

The Two-Stage Framework

  1. Model Specification: The framework defines a substantive model (the core relationship, e.g., Emotion $\to$ Likes) and an imputation model (using cheap auxiliary features $W$ to predict the expensive binary covariate $Z$).
  2. Probability Imputation: Instead of using a "hard" classification threshold, the method imputes missing binary features using the predicted probabilities from a logistic regression. This approach preserves the inherent uncertainty of the prediction model.
  3. Weighted Estimator: To ensure robustness, the authors developed a "Further Improved Estimator." This is a weighted average of the pilot-only results and the imputed results, optimized to minimize the variance of the final estimates.
  4. Asymptotic Theory: The paper establishes the mathematical conditions under which these estimators are consistent and follow a normal distribution, enabling the calculation of valid p-values and confidence intervals.

Application: Emotion Recognition in Live Streaming

The method was validated using a massive real-world dataset from DouYin (TikTok China), consisting of 2,744,173 video clips (5 seconds each) from over 3,000 live streams.

Live streaming interface elements used for data collection and emotion recognition analysis.

Data Components

  • Pilot Sample: 6,568 clips were manually coded by experts for "Valance" (positive vs. non-positive) and "Arousal" (strong vs. weak emotion).
  • Auxiliary Features: 7-dimensional vectors extracted via Mel spectrograms and a VGG16 neural network were used as the cheap predictors for the imputation model.
  • Outcome Metrics: The study focused on cumulative likes and other engagement indicators visible on the streaming interface.

Key Results and Insights

  • Statistical Efficiency: In both simulations and real-world tests, the proposed imputed estimator consistently outperformed the pilot-only estimator, showing significantly lower Mean Squared Error (MSE) and smaller standard errors.
  • Impact of Emotion: The analysis revealed that both positive valence and strong arousal have a statistically significant positive effect on the number of likes. The total effect of emotion accounts for approximately 13.7% of the standard deviation in the response variable.
  • Noise Advantage: The imputed estimator is particularly powerful when the random noise in the main regression is large. In such scenarios, the benefit of the increased sample size from imputation far outweighs the error introduced by the prediction model.
  • Handling Imbalance: The paper provides specific theoretical adjustments for cases where the binary feature is rare, ensuring the method remains robust even for "needle in a haystack" scenarios.

Conclusion and Future Work

This research provides a robust solution for scaling regression analysis when labels are expensive. By using continuous probabilities for imputation and a weighted estimation strategy, the method avoids the biases of naive machine learning applications while maximizing the utility of large, unlabeled datasets.

Future research directions include extending the theory to high-dimensional features where the number of predictors grows with the sample size, and adapting the substantive model for Generalized Linear Models (GLMs) to handle a wider variety of outcome types, such as count data.

Summary updated

Abstract

In the flourishing live streaming industry, accurate recognition of streamers' emotions has become a critical research focus, with profound implications for audience engagement and content optimization. However, precise emotion coding typically requires manual annotation by trained experts, making it extremely expensive and time-consuming to obtain complete observational data for large-scale studies. Motivated by this challenge in streamer emotion recognition, we develop here a novel imputation method together with a principled statistical inference procedure for analyzing partially observed binary data. Specifically, we assume for each observation an auxiliary feature vector, which is sufficiently cheap to be fully collected for the whole sample. We next assume a small pilot sample with both the target binary covariates (i.e., the emotion status) and the auxiliary features fully observed, of which the size could be considerably smaller than that of the whole sample. Thereafter, a regression model can be constructed for the target binary covariates and the auxiliary features. This enables us to impute the missing binary features using the fully observed auxiliary features for the entire sample. We establish the associated asymptotic theory for principled statistical inference and present extensive simulation experiments, demonstrating the effectiveness and theoretical soundness of our proposed method. Furthermore, we validate our approach using a comprehensive dataset on emotion recognition in live streaming, demonstrating that our imputation method yields smaller standard errors and is more statistically efficient than using pilot data only. Our findings have significant implications for enhancing user experience and optimizing engagement on streaming platforms.
BibTeX
@article{lin2025statistical,
  author = {Ziqian Lin and Danyang Huang and Ziyu Xiong and Hansheng Wang},
  title = {{Statistical Inference for Regression with Imputed Binary Covariates with Application to Emotion Recognition}},
  journal = {The Annals of Applied Statistics},
  year = {2025},
  doi = {10.1214/24-aoas1961},
}
Figure