Conversational Decision-Making Model for Predicting the King’s Decision in the Annals of the Joseon Dynasty

JinYeong Bak, Alice Oh

TL;DR

This paper introduces a novel Conversational Decision-Making Model (CDMM) that leverages hierarchical RNNs with attention and speaker embeddings to predict historical royal decisions from the Annals of the Joseon Dynasty. The model significantly outperforms baselines, demonstrating the crucial role of speaker identity in understanding and predicting group decision-making.

What Problem Does This Paper Address?

Understanding how group decisions are made, especially in historical contexts, is challenging. While previous work has analyzed meeting recordings to identify decisions, these often lack the complexity and scale of historical archives. This paper tackles the problem of predicting a leader's decision from a rich, longitudinal corpus of historical meeting records – the Annals of the Joseon Dynasty (AJD). The AJD presents a unique challenge due to its sheer volume (spanning over 400 years) and the nuanced dynamics of royal court discussions, where not only the content but also the speakers involved likely influence the final outcome. The authors aim to formalize this problem and develop a model that can capture these subtle influences.

Methods

The core of the paper is the Conversational Decision-Making Model (CDMM). It builds upon the Hierarchical Attention Network (HAN) architecture, which is effective for document classification by attending to words within utterances and then to utterances within a document. However, CDMM adapts this by:

  • Utterance-level encoding: Instead of sentence-level, it processes utterances directly.
  • Speaker Information Integration: This is the key innovation. CDMM explicitly incorporates speaker identity into the model.
  • Speaker Embeddings: They generate speaker embeddings using a co-occurrence document network. This network connects individuals who appear together in the same article within the AJD. The node2vec algorithm is then used to learn vector representations for each speaker based on this network. This captures implicit relationships and community information among individuals.
  • GRU with Speaker Input: The utterance encoder (a bi-directional GRU) takes both the utterance representation and the speaker embedding as input, allowing it to learn how different speakers contribute to the decision.

Datasets

The primary dataset used is the Annals of the Joseon Dynasty (AJD). The authors curated a dataset from this historical corpus, focusing on meeting records involving kings and their officials. They filtered for articles with at least three speakers and identified the king's final decision. The final dataset comprises records from fifteen kings with over 200 decision-annotated articles each, totaling 13,216 articles and 95,615 utterances. This is a newly constructed benchmark for historical decision-making analysis.

Key Insights and Ideas

  • Speaker Identity Matters: The most significant insight is that knowing who is speaking is crucial for predicting decisions, even more so than just the words spoken. CDMM with speaker embeddings significantly outperforms models without this information.
  • Historical Context is Rich: The AJD provides a unique and valuable dataset for studying group decision-making dynamics, offering a long-term perspective not available in modern meeting corpora.
  • Co-occurrence Networks Capture Social Structure: Building speaker embeddings from co-occurrence networks effectively captures the social and organizational structure of the Joseon court, which in turn aids in decision prediction.
  • Attention Reveals Reasoning: The attention mechanism in CDMM not only helps predict decisions but also provides interpretable insights into why certain words and speakers are important for specific decision types.

Key Results

  • CDMM Dominance: CDMM with pre-trained speaker embeddings achieved the highest macro-averaged F1 score of 0.456, outperforming all other methods, including baselines like Naive Bayes, SVM, and fastText.
  • Speaker Information Boost: Even randomly initialized speaker embeddings improved CDMM's performance, highlighting the inherent value of speaker information. SVM and Naive Bayes also saw performance gains when speaker features were added.
  • Interpretability: Analysis of attention weights showed that specific words (e.g., "Wish to do," "Okay") have different importance for different decision classes, and this importance varies based on the speaker. For instance, the word "Okay" from a king might signal acceptance, while from an official, it might be interpreted differently by the model.

Limitations and Future Work

  • Limited Scope of "Decision": The paper focuses on classifying explicit decisions into six categories. Decisions that are postponed or implicitly made are categorized as "Discuss," which might be a simplification.
  • Data Curation Challenges: While the AJD is rich, extracting and annotating decisions is a complex process. The authors acknowledge that some articles might not have explicit decisions recorded.
  • Future Directions: The authors suggest applying CDMM to other group dynamics analyses and exploring its use in understanding behavior in different organizational structures. They also highlight the potential for using this approach for more general historical text analysis.

Summary updated

BibTeX
@inproceedings{bak2018conversational,
  author = {JinYeong Bak and Alice Oh},
  title = {{Conversational Decision-Making Model for Predicting the King’s Decision in the Annals of the Joseon Dynasty}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure