TL;DR
This paper introduces PoliCorp, a web-based platform designed to make parliamentary debate transcripts easily searchable and analyzable for researchers. It addresses the challenge of accessing and processing vast amounts of political text data by offering an intuitive interface for querying, filtering, and downloading custom subcorpora without requiring programming skills.
What Problem Does This Paper Address?
Analyzing political discourse is crucial for understanding agenda-setting, ideological positions, and power dynamics within governments. However, accessing and working with parliamentary data presents several hurdles. While many countries offer open data initiatives for parliamentary records, this raw data often requires significant preprocessing and specialized analytical skills. Existing platforms may be subscription-based, lack standardized annotations, or embed data within proprietary software, limiting accessibility and usability for researchers. This paper aims to bridge this gap by providing a user-friendly, structured, and searchable platform for political text corpora.
Methods
PoliCorp processes raw XML data from parliamentary transcripts through a multi-stage pipeline. This involves:
- Preprocessing: Converting XML data into a unified database schema, including sentence splitting and parsing for specific linguistic features like "calls to order."
- Feature Extraction: This stage includes:
- Interjection and Call to Order Identification: Developing rule-based systems to identify "calls to order" (which are not explicitly marked in raw data) and leveraging existing annotations for interjections.
- Named Entity Recognition (NER): Integrating experimental NER models (Legal German NER and German NER) to identify entities within the text.
- Topic Classification: Applying a BERT-based model to classify speech contributions into 21 distinct topics relevant to German parliamentary discourse, with a special category for "Presidency Action."
- Postprocessing: Disambiguating speaker names and political party names using rule-based approaches and pattern matching to handle inconsistencies in the raw data.
- Indexing: Utilizing Elasticsearch for efficient web-based searching of the processed data.
The frontend is built with Express.js and Pug, offering both basic and advanced search functionalities with logical operators and dynamic query translation to Elasticsearch.
Datasets
The primary dataset used is the GermaParl corpus, which contains transcripts of German Bundestag plenary debates from September 7, 1949, to September 7, 2021. Additional data for speeches published after September 7, 2021, was sourced from the Bundestag Open Data project. The paper does not introduce new benchmark datasets but rather focuses on processing and making existing ones accessible.
Key Insights and Ideas
- Bridging the Gap: PoliCorp effectively democratizes access to complex political data by abstracting away the technical complexities of data processing and search.
- Beyond Sentiment: The focus on "calls to order" as a marker of disruptive language and incivility offers a novel approach to analyzing parliamentary discourse, moving beyond traditional sentiment analysis.
- Longitudinal Topic Analysis: The topic classification allows for the visualization of the evolution of key debate topics over decades, revealing shifts in political priorities (e.g., the increasing relevance of the "environment" topic).
- User-Centric Design: The platform's development was guided by user feedback, ensuring that its features align with the needs of political scientists and researchers.
Key Results
- The PoliCorp platform currently hosts over 1 million speech contributions from the German Bundestag, spanning 76 years of proceedings.
- The platform enables users to create custom subcorpora by defining search parameters and download them in JSON format for further analysis.
- User query statistics show a high frequency of searches for specific legislative periods, political parties (CDU, SPD, DIE LINKE), and prominent politicians (Merkel).
- Experimental features like NER and topic classification are integrated, providing additional layers of analysis.
Limitations and Future Work
- Prototype Status: PoliCorp is a demonstration version currently under development.
- Experimental Features: Topic classification and NER outputs are generated automatically and may contain inaccuracies, requiring users to verify critical information.
- Domain Specificity: The resource is primarily intended for users engaged in political science research.
- Future Work:
- Implementation of a toxicity detection module.
- Linking transcripts with corresponding video recordings.
- Integration of additional corpora (e.g., StateParl) for consistent processing and comparative analysis.
- Enhancement of analysis tools (descriptive statistics, visualization, cross-corpus comparison).
- Expansion of download formats.
- Broader evaluations through surveys and workshops.
- Making the tool's source code publicly available.
Summary updated