TL;DR
This paper introduces Conversational Information Gain (CIG), a novel framework to measure the substantive informational progress of public deliberations. By tracking an evolving "semantic memory" of a conversation using Large Language Models (LLMs), the system extracts and consolidates atomic claims to evaluate how much each utterance advances collective understanding.
What Problem Does This Paper Address?
Evaluating the quality of public debates or community forums has traditionally relied on surface-level proxies like civility, formal argument structures, or utterance length. However, these metrics can be deceptive: a highly polite conversation can result in complete stagnation, while an informal, messy exchange can yield breakthroughs.
Existing NLP metrics fail to capture whether a conversation is actually getting anywhere informationally. The authors address this gap by defining and operationalizing CIG—measuring the actual change an utterance induces in the collective knowledge base of the participants.
Methods
The authors break down CIG into three core dimensions:
- Novelty: Is this information new to the conversation?
- Relevance: Does it align with the core topic or goal?
- Implication Scope: How far do its implications generalize beyond the immediate speaker or case?
To automate this, they build an entailment-based Semantic Memory Pipeline:
- Claim Extraction: An LLM breaks down complex, context-dependent utterances into distinct, self-contained atomic propositions.
- Multi-Party Memory Consolidation: New claims are compared against retrieved existing memories using a bidirectional Natural Language Inference (NLI) judge. The system then executes a deterministic action:
ADD(new info),UPDATE(refines or contradicts existing info), orNONE(redundant info). - LLM Evaluation: Using the consolidated memory as context, an LLM judge scores the utterance's CIG and its three sub-dimensions on a 1–4 scale.
Datasets
The framework was validated on 80 dialogue segments curated from two distinct moderated settings:
- Intelligence Squared (INSQ): A dataset of competitive, structured TV debates featuring domain experts.
- FORA Corpus: A dataset of collaborative, brainstorming community discussions regarding city planning.
Human annotations were gathered via Prolific to establish a gold standard, achieving moderate-to-high inter-annotator agreement (Krippendorff's $\alpha$ between 0.50 and 0.67).
Key Insights and Ideas
- The "Conjunctive Bottleneck": An unsupervised aggregation analysis revealed that an utterance's overall information gain is effectively capped by its weakest dimension. If a comment is highly novel but completely irrelevant, its CIG is low. Mathematically, a
minoperator combining the dimensions predicted human scores best.
- The Nuance of "Scope": Counterintuitively, a broader "Implication Scope" (e.g., stating a universal principle) does not monotonically increase perceived informativeness. Low-scope, deeply personal testimonies can be highly informative if they provide concrete, diagnostic facts.
- Memory Dynamics > Surface Heuristics: Simple signals derived from the semantic memory state (like the count of claim updates) track human judgments of informativeness far better than traditional text features like TF-IDF or token length.
Key Results
- LLM vs. Human Consensus: GPT-5 achieved a Mean Absolute Error (MAE) of 0.457 on INSQ and 0.520 on FORA when predicting overall CIG. This matches or outperforms the human leave-one-out (LOO) baseline, proving that LLMs can reliably automate this evaluation.
- Feature Correlation: Relevance-gated memory updates (
Memory changes (Relv+)) achieved the highest correlation with human CIG ratings ($|r| = 0.726$), outperforming token length ($|r| = 0.697$) and token surprisal ($|r| = 0.678$). - Downstream Application (Moderator Dynamics): The authors used CIG to analyze how different moderation styles affect a room. In competitive debates (INSQ), sharp moderator interventions ("Confronting") trigger immediate, high-CIG spikes from participants. In collaborative settings (FORA), information gain accumulates gradually and is distributed across subsequent turns.
Limitations and Future Work
While promising, the authors note several limitations:
- Information Constraints: Human annotators were provided with a summarized memory context rather than the full transcript history, which might bias ratings toward what the summary captured.
- Error Propagation: Because the pipeline is sequential, mistakes made during the initial LLM claim extraction or NLI matching can cascade into downstream memory dynamics.
- Scope of Evaluation: The study was restricted to moderated, English-language dialogues and intentionally excluded highly polarized or sensitive topics.
Future work includes integrating advanced memory architectures to improve long-context tracking and applying CIG as a reward signal for training deliberation-support AI systems.
Summary updated