Abstract
This study utilizes machine learning to analyze two decades of U.S. Congressional speeches, demonstrating that a senator's party affiliation can be predicted with up to 99.8% accuracy based solely on word choice. By developing text-based "ideology scores" and comparing them to actual roll-call voting records (DW-NOMINATE), the research quantifies the gap between public rhetoric and legislative action. The findings reveal that politicians strategically use non-binding speeches to align themselves with party mainstream positions, even when their voting records diverge, providing a data-driven metric for political "untrustworthiness."
Introduction
The discrepancy between what politicians say and how they vote is a foundational concern in political science. While speeches are public and highly visible, they are non-binding; conversely, votes are recorded positions that directly influence policy but may be less accessible to the average constituent. Historically, measuring the divergence between these two forms of political expression required labor-intensive manual analysis.
This paper addresses the challenge of quantifying political ideology and trustworthiness at scale. By applying machine learning to the massive corpus of the U.S. Congressional Record from 1995 to 2015, the study seeks to determine if a "text-based" version of traditional voting-based ideology scores can be created. This allows for a direct, quantitative comparison between a politician's public rhetoric and their legislative behavior.
Methodology
The research treats political ideology as both a classification and a regression problem, employing a bag-of-words approach to analyze the frequency of specific terms in the U.S. Congressional Record. The methodology consists of four primary stages:
- Classification Models: Four classical machine learning models—Decision Trees (DT), Naïve Bayes (NB), Support Vector Machines (SVM), and Lasso-penalty Regression (LR)—were trained to predict a senator's party affiliation based on their speech patterns.
- Ideology Embedding: Using the probabilities generated by the Lasso Regression model, the author constructed a one-dimensional "ideology score" derived purely from text.
- Comparative Analysis: These text-based scores were benchmarked against DW-NOMINATE scores, the gold-standard metric in political science for measuring ideology based on roll-call voting records.
- Temporal Testing: To account for the evolution of political language, models were trained on specific sessions of Congress and tested on others to measure "out-of-sample" performance over time.
Key Results
High-Precision Classification
The study achieved remarkable success in identifying partisan identity through language. Using Lasso Regression on the full dataset, the model predicted a senator's party with 98.3% accuracy. When the model first predicted a senator's DW-NOMINATE score (voting behavior) and then used that prediction to classify their party, the accuracy reached a staggering 99.8%.
The Evolution of Political Language
The analysis found that classifier accuracy significantly declines as the "temporal distance" between the training data and the test data increases. This confirms that Congressional vocabulary is a moving target; a model trained on 1990s rhetoric struggles to categorize 2010s politicians because the specific terms and phrases used to signal polarization shift over time.
Strategic Rhetoric and the "Party Line"
Regression analysis revealed a statistically significant correlation (at the 1% level) between the gap in a senator's speech and their votes and their distance from the party mean. The data suggests that when a senator’s rhetoric diverges from their voting record, it is often a strategic move to appear closer to the "party line" or the perceived expectations of their base. Essentially, politicians use non-binding speech to smooth over idiosyncratic or extreme voting records.
Discussion and Implications
One of the most striking findings is the incumbency gap: experienced politicians tend to exhibit larger discrepancies between their speech and their votes. This suggests that as politicians gain job security, they may become more adept at maintaining a divergence between their public persona and their legislative behavior.
While the study provides a robust framework for quantifying political trustworthiness, it faces certain limitations. The reliance on classical machine learning and bag-of-words models means the analysis does not capture the complex semantic context that modern neural networks might identify. Furthermore, the data is limited to floor speeches; incorporating social media or campaign advertisements could provide a more comprehensive view of political signaling.
Conclusion
By bridging the gap between natural language processing and political science, this research offers a powerful tool for monitoring legislative behavior. The ability to quantify the difference between rhetoric and action provides a new lens through which to view political accountability, suggesting that "untrustworthiness" is not merely a subjective observation but a measurable strategic behavior in modern governance.
Summary updated