Analysis of State-Level Legislative Process in Enhanced Linguistic and Nationwide Network Contexts

Maryam Davoodi, Dan GoldwasserNAACL 2024 · DOI

TL;DR

This paper introduces a novel framework for analyzing the complex, state-level legislative process across the US. By leveraging Generative Large Language Models (LLMs) to normalize inconsistent state policy language and building a nationwide legislative graph enriched with diverse contextual data (like interest groups and political stances), the authors enable more accurate prediction of legislative outcomes and legislator behavior.

What Problem Does This Paper Address?

Analyzing state-level legislation is crucial due to its significant impact on daily life, yet it's hampered by several challenges. State bills are notoriously inconsistent in language, format, and complexity across different states, making cross-state policy analysis difficult. Existing NLP/ML studies often focus on the federal level, overlook crucial state-specific contextual information (like local lobbyists), use low-quality data, or rely on traditional architectures. This paper aims to bridge this gap by creating a unified, context-aware framework for understanding the nuances of state legislative processes nationwide.

Methods

The core of the framework is a nationwide, multi-relational, text-attributed legislative graph. This graph is built upon four key principles:

  1. Nationwide Scope: It encompasses all 50 US states, unlike many prior studies.
  2. Contextual Enrichment: It incorporates diverse entities and relations, including legislators, bills, interest groups (IGs), and Political Courage Tests (PCTs).
  3. LLM-Powered Normalization: Generative LLMs (like OpenAI's GPT) are used to normalize inconsistent state legislative language, generate consistent summaries, enrich legislator data, and identify potential bill winners/losers. This addresses the verbosity, complexity, and cross-state variations in bill text.
  4. Joint Text and Graph Embedding: A novel architecture combines LLM-based text embeddings with a heterogeneous Graph Neural Network (GNN), specifically RGCN, to jointly learn representations for nodes and relations. This approach aims to preserve linguistic context while capturing relational information, and is designed to handle longer texts and unseen entities more effectively.

The framework supports three key prediction tasks: roll-call (vote) prediction, PCT prediction, and IG score prediction.

Datasets

The primary dataset used is derived from Vote Smart (VoteSmart, 2023), a non-profit organization providing unbiased information on US political figures. This includes:

  • 7K pieces of legislation from 2019-2022.
  • 16.9K legislative actions.
  • Information on 7237 legislators.
  • Data on 1421 Interest Groups (IGs).
  • 61K IG scores (ratings of legislators).
  • 2216 unique PCT questions answered by 902 legislators.

Additionally, the authors generated LLM-extracted bill highlights and winner/loser relations, which were compared against ground-truth data from Vote Smart researchers and full bill texts.

Key Insights and Ideas

  • LLMs are crucial for state-level policy analysis: The inherent linguistic inconsistencies and verbosity of state legislation make LLMs indispensable for normalization and summarization, enabling meaningful cross-state comparisons.
  • Context is king (and queen): Incorporating diverse contextual information like interest group influence and political stances (PCTs) significantly improves the understanding of legislative behavior beyond just bill text.
  • Joint text and graph models offer synergistic benefits: Combining LLM-based text embeddings with GNNs allows the model to leverage both the rich semantic content of legislative documents and the relational structure of the political landscape.
  • Predicting legislator behavior is multifaceted: The paper demonstrates that understanding legislator votes requires considering not just the bill itself, but also their alignment with interest groups and their stances on controversial issues.

Key Results

  • LLM optimization significantly improves performance: Activating LLM-based optimization on the legislative graph improved all baseline models and their own, leading to a 3-point increase in F1 score for the Ada model on the roll-call task.
  • Joint model outperforms baselines: The proposed joint text and graph model achieved an F1 score of 77.8 on the roll-call prediction task, outperforming leading individual text or graph models by 2.3 and 4.2 points, respectively.
  • Contextual information is vital: Ablation studies showed that removing IG relations decreased the F1 score by 2.1 points, while excluding PCT relations reduced it by 1.6 points, highlighting the importance of these contexts.
  • Bill highlights are more effective than titles/synopses: Using researcher-crafted bill highlights improved performance by up to 3.5 F1 points compared to official titles and synopses.
  • LLM-generated highlights are largely accurate: In a small-scale evaluation, GPT-generated highlights were found to be accurate in 95% of sampled bills, sometimes even surpassing researcher-written highlights.

Limitations and Future Work

  • Model-Specific Challenges:
    • Scaling Training/Inference: The joint LLM+GNN model is computationally intensive. Future work will explore graph knowledge distillation (KD) and more efficient distributed training platforms.
  • Choice of LLM/GNN: While OpenAI GPT/Ada and RGCN were chosen for performance, exploring open-source models (Phi-2, Llama-2) is a future direction to reduce costs.
  • Handling Unseen Entities: While RGCN uses inductive inference, improving the handling of unseen nodes through LLM-based entity alignment and further fine-tuning is planned.
  • Data-Specific Limitations:
    • Focus on Floor Votes: The study primarily focused on "floor vote" or "roll call vote," excluding other bill actions like introduction or committee review.
  • Abstain Votes Ignored: Abstain votes were not considered due to their rarity, but predicting them is identified as an interesting future problem.
  • Data Source Bias: Reliance on Vote Smart data might introduce biases.
  • Copy-cat Bills: While the dataset focuses on novel legislation, the potential impact of copy-cat bills and the development of tasks to identify them are future considerations.
  • Future Work:
    • Explore larger open-source LLMs (e.g., 70B Llama-2) for summarization.
  • Investigate the efficiency of open-source models as alternatives to OpenAI Ada for embedding.
  • Develop tasks to identify and classify copy-cat bills.
  • Explore more sensitive classification tasks, like predicting gender-charged voting splits.
  • Examine the potential biases introduced by GPT-generated summaries.

Summary updated

BibTeX
@inproceedings{davoodi2024analysis,
  author = {Maryam Davoodi and Dan Goldwasser},
  title = {{Analysis of State-Level Legislative Process in Enhanced Linguistic and Nationwide Network Contexts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.411},
}
Figure