TL;DR
This paper introduces a massive dataset of 70,509 literary social networks extracted from multilingual fiction and non-fiction texts, enabling large-scale analysis of social structures in literature. By leveraging a novel prompt-based approach with a large language model (LLM), the authors automate a previously manual and time-consuming process, opening new avenues for humanities and social science research.
What Problem Does This Paper Address?
Literary scholars have long been fascinated by how novels depict social interactions and relationships. However, analyzing these social worlds at scale has been a significant challenge. Traditional methods of manually extracting social networks from texts are incredibly labor-intensive and simply don't scale to the vast number of literary works available. Existing automated methods, often relying on supervised machine learning, have been limited in their accuracy and scalability, restricting studies to relatively small corpora. This limits the statistical significance and generalizability of findings about literary social structures.
Methods
The core innovation lies in their method for extracting social networks. Instead of traditional supervised learning, they adapt a prompt-based approach for Google's Gemini 1.5 Flash LLM. This involves crafting a specific prompt that instructs the LLM to identify characters and their relationships within a given text, outputting the results in a structured JSON format. This approach leverages the LLM's ability to understand complex linguistic nuances and its large context window (1 million tokens) to process entire literary works. The structured output, enforced by a JSON Schema, ensures consistency and comparability across the vast number of texts. They validate this approach by comparing its output to a smaller, manually annotated dataset, demonstrating promising accuracy.
Datasets
- Project Gutenberg (PG) Corpus: The primary source of literary texts, providing over 70,000 volumes.
- MultiHATHI: An extended multilingual edition of the HathiTrust Digital Library catalog, used to enrich PG texts with metadata like publication date, language, and fiction/non-fiction status. This process yielded 33,919 well-documented texts.
- Massey et al. (2015) Dataset: A smaller, manually annotated dataset of 2,170 character relationships from 109 fictional narratives, used for validating the LLM-based extraction method.
Key Insights and Ideas
- LLMs as Scalable Annotators: The paper demonstrates that LLMs, when guided by carefully crafted prompts and structured output constraints, can effectively automate complex annotation tasks like social network extraction at an unprecedented scale. This opens the door for similar applications in other areas of digital humanities.
- Fiction vs. Non-Fiction Social Structures: A significant finding is the difference in network properties between fiction and non-fiction. Fictional networks tend to be smaller, more clustered, and have fewer disconnected components, suggesting more tightly-knit social worlds compared to non-fiction.
- Relationship Type Differences: Fiction leans more heavily on social and familial relationships, while non-fiction shows a greater proportion of professional relationships, aligning with the idea that fictional characters might inhabit more private or domestic spheres.
- Community Structure: Non-fiction texts exhibit a wider range of community counts, while fiction texts show a more consistent number of communities, further supporting the notion of more structured social environments in fiction.
Key Results
- Dataset Size: The paper releases a dataset of 70,509 high-quality social networks.
- Metadata: Extended metadata is provided for ~30,000 works (22,015 non-fiction, 7,331 fiction).
- Validation Accuracy: The LLM-based pipeline achieved 81% accuracy for relationship valence and 74% for "fine category" annotations when compared to human annotations. Accuracy for "coarse category" was lower (55%), leading the authors to use the coarse labels derived from the fine categories.
- Network Size: Non-fiction networks are significantly larger on average (22.14 nodes vs. 42.69 nodes; 27.42 edges vs. 42.91 edges).
- Clustering: Fictional networks are more clustered, with significantly fewer disconnected components (2.20 vs. 5.12) and higher mean transitivity (0.22 vs. 0.12).
- Community Counts: Non-fiction networks have a higher mean number of communities (11.28 vs. 8.65).
Limitations and Future Work
- Data Bias: The textual data is predominantly sourced from European authors, with English being the most dominant language. This limits the generalizability to other cultural contexts.
- LLM Context Window and Output Limits: Gemini 1.5 Flash's context window (1 million tokens) meant that approximately 374 volumes could not be fully processed. Additionally, the API's output limit (8,000 tokens) may result in incomplete networks for some very large texts.
- Narrative Perspective: The current method doesn't account for how different narrative perspectives might influence the representation of social networks within a single text.
- Future Work: The authors highlight the need for better, open-source, and locally runnable alternatives to proprietary LLMs, which often have limitations in interpretability, token length, and content filtering. They also emphasize the need for improved evaluation methods for social network extraction. Future research should explore methods for assessing the validity of extracted social networks and incorporating narrative perspective.
Summary updated