Introduction
Traditional Information Retrieval (IR) systems typically operate on a "one-and-done" query model. However, Conversational Search Systems (CSS) are inherently sequential and interactive, making evaluation and user modeling significantly more complex. A poor response in one turn can fundamentally alter a user's strategy in the next. To improve these systems, researchers often employ User Simulators (US), but these simulators frequently lack a grounded understanding of the psychological triggers behind user actions, such as ending a session or pivoting to a new topic.
This research explores the application of "priming effects"—a psychological phenomenon where past experiences unconsciously influence current behavior—to model user actions in CSS. By analyzing repetition, semantic similarity, and sentiment across multiple dialogue datasets, the study demonstrates that these stimuli can accurately predict when a user will stop a conversation or ask a follow-up question.
Quantifying Priming Stimuli
The authors developed a comprehensive framework to quantify 270 different "stimuli" based on three distinct types of priming:
- Repetition Priming: Measures the influence of repeating specific words or noun phrases from previous turns.
- Semantic Priming: Evaluates how semantically related words (calculated via Word2Vec) "awaken" concepts in the user's mind.
- Affective Priming: Analyzes how the emotional tone (polarity and subjectivity) of system responses affects user reactions.
As shown in the framework above, these stimuli are calculated across three dimensions: Depth (how many turns back the influence persists), Scope (whether stimuli originate from user queries, system replies, or both), and Aggregation (whether the effect is summed, averaged, or based on a maximum value).
Experimental Setup and Datasets
The study utilized six diverse conversational datasets to ensure generalizable findings:
- TopiOCQA & FAITHDIAL: Focused on free-form agent responses.
- QReCC & ORConvQA: Large-scale conversational QA datasets.
- TREC CAsT 2021 & EvalCran: Focused on passage retrieval.
- AllMix: A balanced benchmark created by the authors by sampling instances from the above datasets to provide a generalized view.
The researchers employed Spearman’s correlation to identify alignment between stimuli and actions, followed by training Logistic Regression (LR) models to predict user actions in real-time.
Key Insights
- Sentiment as a Termination Signal: Affective priming (sentiment-based stimuli) showed a strong negative correlation with conversation length. Subjective or highly polarized system responses often serve as a trigger for users to end a session.
- The Power of the "Mean": Logistic Regression models performed better when using the average (mean) of stimuli over several turns rather than the total sum or a single maximum "spike." This suggests that the cumulative, sustained environment of a conversation influences users more than any single outlier turn.
- Feature Consistency: Stimuli based on the same underlying feature (e.g., different versions of semantic similarity) played similar roles across different actions, suggesting user behavior is influenced by the nature of the information rather than the specific calculation method.
Results and Predictive Performance
The predictive power of priming-based models varied significantly depending on the specific user action being modeled.
As detailed in the table above, the models achieved high performance in predicting Stopping actions, reaching an AUC of 0.835 on the AllMix dataset and near-perfect scores on FAITHDIAL. Predicting Following up actions showed moderate success (AUC of 0.698 on ORConvQA). However, Switching Topic remains a significant challenge, with the model achieving an AUC of only 0.476—essentially no better than random guessing. This suggests that topic switching is driven by complex cognitive factors outside the immediate linguistic context of priming.
Conclusion and Future Work
While priming effects offer a robust lens for predicting conversation termination and follow-up intent, they struggle to capture the nuances of topic switching. Future research may need to incorporate other forms of influence, such as social or procedural priming. Additionally, the scarcity of explicitly labeled data for "following up" and "switching topics" remains a hurdle; more robustly labeled datasets will be essential to confirm these findings across broader conversational domains.
Summary updated