Conversational Decision Making Model for Predicting King's Decision in the Annals of the Joseon Dynasty

JinYeong Bak, Alice Oh

Introduction

Understanding group decision-making is a complex task in social psychology and management. While modern meeting datasets exist, they are often limited in scope or scale. This research, presented at EMNLP 2018, leverages the Annals of the Joseon Dynasty (AJD)—a massive historical record of royal discussions from 1392 to 1910—to model how leaders arrive at final decrees based on group deliberation.

The Annals of the Joseon Dynasty (AJD)

The AJD provides a unique longitudinal corpus where the King discusses state affairs with various officials. Each entry typically includes the time, a title, factual background, and a transcript of the dialogue between the King and his advisors, culminating in a final royal decision.

An annotated example of a record from the Annals of the Joseon Dynasty showing the dialogue structure and the final decision.

The researchers curated a subset of 13,216 articles involving 15 kings and over 4,500 unique historical figures. Decisions were categorized into six labels: Order, Approve, Disapprove, Accept, Reject, and Discuss.

The Conversational Decision-Making Model (CDMM)

To predict these decisions, the authors proposed the Conversational Decision-Making Model (CDMM). The architecture is designed to capture both linguistic cues and the social influence of the participants.

The architecture of the CDMM, featuring hierarchical encoders for words and utterances, integrated with speaker embeddings derived from social networks.

The model consists of three main pillars:

  1. Hierarchical RNN with Attention: A word-level encoder identifies key terms within an utterance, while an utterance-level encoder tracks the sequence of the entire conversation to determine which turns were most influential.
  2. Speaker Embeddings: To account for the "who" in the discussion, the model incorporates speaker-specific vectors. In historical contexts, the weight of an opinion is inextricably tied to the rank and reputation of the speaker.
  3. Social Network Pre-training: Since historical figures lack standard pre-trained embeddings, the authors used the node2vec algorithm on a co-occurrence network of 4,502 officials to capture political and social proximity.

Key Insights and Results

The inclusion of speaker information proved critical to the model's success. The CDMM with pre-trained speaker embeddings achieved a Macro F1 score of 0.313, nearly doubling the performance of models that ignored speaker identity (0.178). The weighted-average F1 reached 0.456, demonstrating a robust ability to handle the nuances of royal dialogue.

Qualitative analysis of the attention mechanisms showed that the model learned context-dependent meanings. For instance, the word "Okay" has different predictive weights depending on whether it is uttered by the King (predicting an Order) or an official (predicting Acceptance). Furthermore, specific roles like the Chief Secretary or Remonstrators were found to have disproportionate influence on certain types of decisions, reflecting the historical reality of the Joseon court.

Conclusion

The CDMM demonstrates that modeling the social context and hierarchy of speakers is essential for understanding group decision-making. While challenges like class imbalance and archaic language nuances remain, this work provides a robust framework for analyzing historical and modern conversational dynamics, from corporate boardrooms to online community moderation.

Summary updated

BibTeX
@inproceedings{bak2018conversational,
  author = {JinYeong Bak and Alice Oh},
  title = {{Conversational Decision Making Model for Predicting King's Decision in the Annals of the Joseon Dynasty}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure