TL;DR
Local governments make crucial decisions, but understanding public input is challenging due to the sheer volume of meetings and the difficulty of automatically identifying public comments. This paper introduces PUBLICSPEAK, a probabilistic framework that leverages meeting structure and linguistic cues to accurately extract public remarks from local government meetings, significantly outperforming existing methods.
What Problem Does This Paper Address?
Local governments are at the forefront of policy-making on issues affecting communities daily. To ensure democratic participation and accountability, it's vital for citizens to voice their concerns and for this input to be heard and understood. Traditionally, local news outlets played a key role in informing the public and covering these meetings. However, the decline of local journalism has created an information gap. While many local governments now record their meetings and make them available online, manually sifting through hours of video and audio to identify and analyze public comments is a monumental task. Existing automated speech recognition (ASR) tools can transcribe meetings, but they don't distinguish between official statements and public feedback, nor do they easily identify the speakers' roles. This paper addresses the need for a robust, scalable method to automatically extract and identify public remarks from these recordings, enabling better analysis of community priorities.
Methods
The core of PUBLICSPEAK is a probabilistic framework built using Probabilistic Soft Logic (PSL). This approach allows the model to combine different types of information:
- Meeting Structure: The framework leverages the typical structure of government meetings, recognizing that public comments usually occur during specific segments (e.g., "public comments" or "public hearings") and are often preceded by introductions. It models transitions between these sections and the order of utterances.
- Speaker Roles: It distinguishes between government officials and members of the public by analyzing utterance length, frequency, and linguistic patterns.
- Linguistic Signals: The model incorporates language cues, such as common phrases used to signal the start of public comment periods, and uses fine-tuned pre-trained language models (PLMs) for utterance-level analysis.
Crucially, PUBLICSPEAK fuses these signals. It can also operate in a 3-class setting (Public Comment, Public Hearing, Other) to better disambiguate remarks made during hearings from those made during general public comment periods. This probabilistic approach allows for soft constraints, meaning it encodes domain knowledge (e.g., public speakers tend to have fewer, longer remarks than officials) without enforcing rigid rules.
Datasets
The paper utilizes a dataset of local government meeting recordings from seven cities across the United States: Oakland, California; Richmond, Virginia; Seattle, Washington; Ann Arbor, Michigan; Jackson, Michigan; Lansing, Michigan; and Royal Oak, Michigan. These cities were chosen to represent a variety of meeting structures and socio-economic diversity. The data was collected for approximately one year per city, resulting in a combined dataset of around 30,000 remarks. The authors also introduce a pipeline for collecting, processing, annotating, and publishing these public meeting records, aiming to create a continuously updated, open-source dataset.
Key Insights and Ideas
- Structure is Key: While linguistic cues are important, the temporal structure of meetings (when public comments are typically allowed) is a powerful signal for identifying public remarks, especially in cities with less predictable comment periods.
- Probabilistic Fusion Outperforms: Combining structural and linguistic information in a probabilistic framework (PUBLICSPEAK) yields significantly better results than methods relying solely on one type of signal or simpler rule-based approaches.
- Generalization is Achievable: The framework demonstrates a notable ability to generalize to new, unseen cities, outperforming strong baselines even when trained on data from other municipalities. This suggests its potential for broad application across the US.
- Topic Modeling Reveals Local Priorities: Analyzing the extracted public comments reveals distinct topic distributions across cities, highlighting how local events and issues shape community concerns (e.g., police reform in Seattle and Oakland, mayoral elections in Jackson).
Key Results
PUBLICSPEAK consistently outperformed state-of-the-art comparators, including fine-tuned large language models like RoBERTa and GPT-4o, by an average of 10% and up to 40% in some cities. In the "Leave One City Out" (LOCO) evaluation, designed to test generalization, PUBLICSPEAK still significantly outperformed the nearest competitor (RoBERTa) by 13%. The 3-class setting (distinguishing Public Comments from Public Hearings) generally showed better performance than the 2-class setting, indicating the value of explicitly modeling these distinct public speaking opportunities.
Limitations and Future Work
The authors acknowledge several limitations:
- Data Quality: The quality of video and audio recordings can vary significantly, leading to transcription errors and difficulties in identifying speakers.
- ASR Disparities: Existing ASR technologies may exhibit racial disparities, potentially affecting the accuracy of transcripts for certain demographics.
- Generalization Variance: While PUBLICSPEAK generalizes well, its performance can vary across cities, particularly in those with very few public comments (like Richmond).
Future work includes:
- Scaling the Pipeline: Expanding the data collection and processing pipeline to cover more cities across the United States and potentially globally.
- Continuous Publication: Regularly publishing datasets of public remarks to support community organizers, journalists, and researchers.
- Deeper Analysis: Utilizing the extracted public comments for more in-depth analysis of political discourse, community priorities, and the impact of local governance.
Summary updated