Abstract: Chaozhuo Li, Yu Wu, Wei Wu, Chen Xing, Zhoujun Li, Ming Zhou. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.
2016.
The Context Challenge in Chatbots
Chatbots often struggle to decide when they need to look at previous conversation turns to understand a user's message. In any given dialogue, some messages are self-contained (e.g., "What time is it?"), while others are functionally meaningless without the preceding dialogue (e.g., "Why do you think so?").
Current chatbot architectures usually take an "all or nothing" approach: they either ignore context entirely—leading to irrelevant, "robotic" answers—or they process the entire conversation history for every single turn. The latter is computationally expensive and can introduce "noise"; if a user starts a brand-new topic, forcing the model to consider the old topic can lead to a confused response. The challenge is that these messages are often short and informal, and there are no massive, human-labeled datasets that categorize messages by their "context dependency."
Methodology: Weak Supervision and LSTMs
The authors propose a two-step approach that avoids the need for massive human annotation by using the diversity of potential responses as a weak supervision signal.
1. Weak Supervision via Response Diversity
The core insight is that context-dependent messages tend to have a much wider variety of responses in a large dataset compared to independent ones. They measure this diversity using three statistical markers:
- Entropy: How "spread out" the word distribution is in the responses.
- Maximum Mass ($M(P)$): Whether a single word dominates the responses.
- Average Response Length: Context-dependent messages often trigger more varied and longer explanations.
2. The LSTM Classifier
These statistical markers are used to "label" millions of messages automatically. Then, the researchers train a Long Short-Term Memory (LSTM) network to look at a message and predict its context dependency.
As shown in the framework above, the LSTM encodes the message into a vector, which is then passed through a two-layer feed-forward neural network to make the final classification. By training on the text of the message to predict the diversity of its responses, the model learns the underlying semantics that make a message context-dependent.
Evaluation and Key Results
The researchers utilized massive real-world conversation data, including 5.9 million message triples from Twitter and 3.1 million from Weibo. To verify the model, humans labeled 1,000 messages from each platform to create a "gold standard" for evaluation.
The LSTM-based approach significantly outperformed all baselines, including those using manual feature engineering:
- Accuracy on Weibo: 75.6% (vs. 66.8% for the best non-regression SVM).
- Accuracy on Twitter: 73.4% (vs. 65.4% for the best non-regression SVM).
Qualitatively, the model successfully identified that a message like "Yep, I have heard it on Saturday night" is context-dependent, whereas shallow models often mistakenly label it independent due to its length and specific keywords.
Insights and Future Work
This research demonstrates that responses act as a mirror: if a message's responses are highly predictable, it is likely stand-alone; if they are varied, it likely depends on hidden context. By identifying context-independent messages, a chatbot can "reset" its memory, saving computational power and avoiding topic drift.
However, limitations remain. The study simplified "context" to the single immediately preceding sentence, whereas real human dialogue often stretches back several turns. Furthermore, the model focuses on linguistic context and does not yet account for physical location or shared real-world knowledge. Future work will look toward extending these context windows and further automating the threshold tuning process.
Summary updated