Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models

Varun Kumar, Alison Smith-Renner, Leah Findlater, Kevin D. Seppi, Jordan L. Boyd-Graber

Abstract

Human-in-the-Loop Topic Modeling (HLTM) allows users to refine automated text analysis by providing direct feedback. However, existing evaluations often prioritize final topic coherence over the model's responsiveness to user intent. This paper introduces a framework to measure "user control"—the degree to which a model follows specific instructions. Through simulations involving "Good" and "Random" users, we compare three HLTM implementations: informed priors with Gibbs sampling (info-gibbs), informed priors with Variational Bayes (info-vb), and constraint-based Gibbs sampling (const-gibbs). Our results reveal a fundamental trade-off: while informed priors offer near-perfect control, constraint-based methods maintain higher data consistency and topic quality, often at the expense of ignoring user demands.

Introduction

Topic models like Latent Dirichlet Allocation (LDA) are essential for navigating large text corpora but frequently generate noisy or incoherent results. HLTM systems aim to bridge this gap by allowing users to intervene—for instance, by moving words between topics or merging related clusters. Despite the proliferation of these systems, a standardized comparison of how well they "listen" to users is missing. If a model ignores a user's request because it contradicts the underlying data distribution, the user experiences a loss of agency. This study addresses this gap by proposing a control metric and evaluating how different inference techniques handle user-driven refinements.

Methodology

We evaluated three HLTM systems across seven common refinement operations, such as adding/removing words and splitting/merging topics.

  1. info-gibbs: Employs informed priors by adjusting Dirichlet hyperparameters ($\alpha$ and $\beta$) within a Collapsed Gibbs Sampling framework.
  2. info-vb: A novel HLTM approach using informed priors with Variational Bayes inference.
  3. const-gibbs: Uses potential functions to penalize states that violate user-defined constraints during Gibbs Sampling.

To ensure objective evaluation, we utilized a dataset of 7,000 news articles from the Guardian API and designed two simulated user types: the "Good User," who leverages ground-truth category labels to improve topics, and the "Random User," who provides arbitrary feedback to test the model's limits of obedience. We introduced a Control Metric, defined as the ratio of the actual change in a word's rank to the expected change. A score of 1.0 indicates perfect adherence to the user's instruction.

The Control vs. Coherence Trade-off

Our analysis identifies a central tension between user will and data evidence. Informed priors act as a "heavy hand," forcing the model to prioritize user instructions even when the data is weak. For example, in the add word task, info-gibbs can elevate a rare word to the top of a topic's distribution. Conversely, constraint-based models like const-gibbs act more like "suggestions." These models are more tethered to the statistical patterns of the text; if a user's request is poorly supported by the data, the model may partially or fully ignore it to maintain coherence.

Experimental Results

The experimental results highlight the performance differences between the three methods across various metrics, demonstrating the divergence between obedience and data fidelity.

Comparison of control scores ($C_{Rand}$, $C_{Good}$) and coherence deltas ($Q_{Good}^*$) across different refinement operations and HLTM methods.

Key Findings:

  • User Control: For the "Good User," informed prior methods (info-gibbs and info-vb) achieved near-perfect control scores ($\approx$ 0.98–1.0). In contrast, const-gibbs showed significantly lower control for operations like reorder word (0.19) and create topic (0.81).
  • Topic Quality: const-gibbs consistently outperformed informed prior methods in terms of coherence improvement ($Q_{Good}^*$). For the remove word task, const-gibbs improved coherence by 5.4, compared to just 1.2 for info-vb.
  • Robustness to "Bad" Feedback: When faced with a "Random User" attempting to create topic with nonsensical seeds, const-gibbs effectively ignored the request (control = 0.08). Meanwhile, info-gibbs obeyed the nonsensical request perfectly (control = 0.98), potentially degrading the model's overall utility.

Conclusion and Future Directions

This research demonstrates that there is no "one-size-fits-all" approach to HLTM. Informed priors are ideal when high responsiveness is required, whereas constraints are "safer" for maintaining data integrity. Future systems should consider hybrid approaches or UI signals that explicitly inform users when their requests conflict with the underlying data. While simulations provide a valuable benchmark, future work must validate these findings with real-world user studies to capture the nuances of human interaction in interactive machine learning loops.

Summary updated

BibTeX
@inproceedings{kumar2019why,
  author = {Varun Kumar and Alison Smith-Renner and Leah Findlater and Kevin D. Seppi and Jordan L. Boyd-Graber},
  title = {{Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1637},
}
Figure