Statistical inference for regression with imputed binary covariates with application to emotion recognition

Ziqian Lin, Danyang Huang, Ziyu Xiong, Hansheng Wang

TL;DR

This paper introduces a principled statistical framework for performing regression when your primary binary predictors (like "is the streamer happy?") are too expensive to label at scale. By using a small "pilot" of hand-labeled data and a large set of cheap auxiliary features (like audio embeddings), the authors provide a method to impute missing values and conduct valid statistical inference with higher efficiency than traditional methods.

What Problem Does This Paper Address?

In modern data science, we often want to understand how a specific human behavior or state affects an outcome. For example: Does a live streamer’s positive emotion lead to more "likes"?

To answer this, you need a regression model where the "emotion" is a covariate ($Z$). The problem? Determining emotion accurately usually requires human experts to watch and code thousands of hours of video. In the paper’s real-world example, they had over 2.7 million 5-second clips. Manually labeling all of them is financially and logistically impossible.

Researchers usually settle for one of two sub-optimal paths:

  1. Pilot Only: Run the regression only on the small subset of data they could afford to label. This wastes the massive amount of unlabeled data available.
  2. Naive Imputation: Use a machine learning model to predict the labels for the rest of the data and treat those predictions as "truth." This often leads to biased results and "shrunken" standard errors that make findings look more certain than they actually are.

Methods

The authors propose a two-stage approach that bridges the gap between these two paths.

  1. The Models: They define a substantive model (the relationship they actually care about, like Emotion $\to$ Likes) and an imputation model (using cheap auxiliary features $W$, like Mel spectrograms from audio, to predict the expensive binary covariate $Z$).
  2. Probability Imputation: Instead of using a "hard" threshold (e.g., if probability > 0.5, label as 1), they impute the missing binary features using the predicted probabilities from a logistic regression trained on a small pilot sample. This preserves the uncertainty of the prediction.
  3. The Weighted Estimator: Recognizing that the imputed estimator might perform poorly if the noise in the data is very low, they develop a "Further Improved Estimator." This is a weighted average of the pilot-only results and the imputed results, optimized using "A-optimality" to minimize the variance of the final estimates.
  4. Asymptotic Theory: Crucially, they don't just provide an algorithm; they prove the math. They establish the conditions under which these estimators are consistent and follow a normal distribution, allowing researchers to calculate valid p-values and confidence intervals.

Datasets

  • Simulated Data: The authors tested their method across various scenarios, including "regular" cases, "highly imbalanced" cases (where the binary feature is very rare), and "highly predictable" cases (where the AI model is nearly perfect).
  • DouYin (TikTok China) Live Streaming: A massive real-world dataset consisting of 2,744,173 video clips (5 seconds each) from 3,262 live streams.
  • Pilot Sample: 6,568 clips were manually coded by experts for "Valance" (positive vs. non-positive) and "Arousal" (strong vs. weak emotion).
  • Auxiliary Features: 7-dimensional vectors extracted via Mel spectrograms and a VGG16 neural network.

Key Insights and Ideas

  • Efficiency Gains: The "aha moment" is that even a mediocre prediction model for the missing labels can significantly improve the precision of your regression. The imputation method yields smaller standard errors than using the pilot data alone.
  • The "Noise" Advantage: Interestingly, the imputed estimator is particularly powerful when the random noise ($\sigma^2$) in the main regression is large. In these cases, the error introduced by imputation is negligible compared to the overall noise, making the extra data points extremely valuable.
  • Handling Rare Events: The paper provides specific theoretical adjustments for "highly imbalanced" data. If you are looking for a "needle in a haystack" feature, the convergence rate of the estimator slows down, and the paper explains exactly how to account for that.
  • Probabilities > Labels: By using continuous probabilities for imputation rather than hard 0/1 labels, the method avoids the bias introduced by arbitrary classification thresholds.

Key Results

  • Statistical Power: In simulations, the proposed imputed estimator consistently outperformed the pilot-only estimator, showing much lower Mean Squared Error (MSE).
  • Real-World Impact: In the DouYin analysis, the method revealed that both positive valance and strong arousal have a statistically significant positive effect on the number of likes.
  • Effect Size: The authors calculated that the total effect of emotion accounts for about 13.7% of the standard deviation in the "likes" response, providing a concrete metric for platform optimizers.
  • Validation: The weighted estimator successfully "picked" the best of both worlds, performing as well as the best individual estimator in every tested scenario.

Limitations and Future Work

  • Fixed Dimensions: The current theory assumes a fixed number of auxiliary features. The authors note that extending this to high-dimensional features (where the number of predictors grows with the sample size) is a necessary next step.
  • Linearity: The substantive model is currently a standard linear regression. Future work could expand this to Generalized Linear Models (GLMs) to handle count data or binary outcomes directly.
  • Imbalance Slowdown: The authors acknowledge that in extremely imbalanced cases, the imputed estimator converges slower than the pilot estimator, suggesting there is still room to optimize how we handle rare binary features.

Summary updated

Abstract

In the flourishing live streaming industry, accurate recognition of streamers' emotions has become a critical research focus, with profound implications for audience engagement and content optimization. However, precise emotion coding typically requires manual annotation by trained experts, making it extremely expensive and time-consuming to obtain complete observational data for large-scale studies. Motivated by this challenge in streamer emotion recognition, we develop here a novel imputation method together with a principled statistical inference procedure for analyzing partially observed binary data. Specifically, we assume for each observation an auxiliary feature vector, which is sufficiently cheap to be fully collected for the whole sample. We next assume a small pilot sample with both the target binary covariates (i.e., the emotion status) and the auxiliary features fully observed, of which the size could be considerably smaller than that of the whole sample. Thereafter, a regression model can be constructed for the target binary covariates and the auxiliary features. This enables us to impute the missing binary features using the fully observed auxiliary features for the entire sample. We establish the associated asymptotic theory for principled statistical inference and present extensive simulation experiments, demonstrating the effectiveness and theoretical soundness of our proposed method. Furthermore, we validate our approach using a comprehensive dataset on emotion recognition in live streaming, demonstrating that our imputation method yields smaller standard errors and is more statistically efficient than using pilot data only. Our findings have significant implications for enhancing user experience and optimizing engagement on streaming platforms.
BibTeX
@article{lin2025statistical,
  author = {Ziqian Lin and Danyang Huang and Ziyu Xiong and Hansheng Wang},
  title = {{Statistical inference for regression with imputed binary covariates with application to emotion recognition}},
  journal = {The Annals of Applied Statistics},
  year = {2025},
  doi = {10.1214/24-aoas1961},
}
Figure