Computational studies of agenda setting, the process by which some issues and framings come to dominate public attention, now rest on long chains of automated steps. A typical study groups posts into events, labels each document's frame with a fine-tuned language model, scores outrage, and feeds the resulting counts into a lagged regression. The published conclusions are substantive claims about society: social media frames Supreme Court rulings more diversely than the press, or Russian state media turns to the United States when the economy falters. Yet nobody checks which step of the chain produces the conclusion. Machine learning learned this lesson the hard way. A question-only bag-of-words model beat full embodied agents on embodied question answering (Blindfold Baselines for Embodied QA), and removing the language model from LLM time-series forecasters cost nothing in nearly every test case (Are Language Models Actually Useful for Time Series Forecasting?).
We propose to carry that discipline over to computational media research, with one important change. The thing we ablate is not a model's accuracy but the paper's scientific conclusion. For each published agenda-setting finding, we replace each component of the pipeline in turn with one of three substitutes: a simpler working version, a "blindfolded" version that sees only shortcut cues such as platform, length, or punctuation, and a placebo input. We then ask whether the substantive effect survives, vanishes, or is reproduced by the blindfold. A finding that the blindfold reproduces is a finding about the measurement instrument, not about media.
Core Claim
Our central hypothesis is concrete and falsifiable. Some reported cross-platform framing differences are partly produced by unequal classifier error across platforms. The Supreme Court study reports frame-classification F1 of about 0.75 on news headlines and about 0.53 on Reddit comments. The same study reports higher framing diversity on Reddit, measured as k-means inertia (how spread out the frame assignments are): 1.15 on Reddit versus 0.71 in news, on affirmative action (Understanding Divergent Framing of the Supreme Court Controversies: Social Media vs. News Outlets). A noisier classifier scatters its predictions across more frames, and that alone raises measured diversity. If we inject label noise into the news predictions at the Reddit error rate and most of the 0.44 inertia gap reappears, then the platform comparison is confounded by measurement quality.
More generally, we define a finding as component-robust when it meets three conditions. It must persist when each module is swapped for a simpler competent substitute. It must disappear under placebo inputs. And it must not be reproduced by blindfolded measurement. We summarize the blindfold test with a reproduction ratio:
$$R = \frac{\Delta_{\text{blind}}}{\Delta_{\text{full}}}$$
Here $\Delta_{\text{full}}$ is the effect size reported with the original pipeline, such as an inertia gap or a lagged regression coefficient. $\Delta_{\text{blind}}$ is the same quantity computed when the measurement module is replaced by a model trained to predict the labels from shortcut features alone. An $R$ near zero supports the substantive reading of the finding; an $R$ near one undermines it.
Relation to Prior Work
The motivation comes from two lines of work. One shows that complex systems often win for reasons other than their advertised component, as in the blindfold and time-series ablations above. The other is the broader critique of "speculation disguised as explanation" and of ignored, well-tuned baselines (Troubling Trends in Machine Learning Scholarship). Both papers ask whether a model's score is earned. We ask the different question of whether a social-science inference is earned.
Within agenda-setting research, the closest existing ablation is SLAP4SLIP (Modeling Ideological Agenda Setting and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity). It compares salience-only, framing-only, and combined graph auto-encoders for predicting links between subreddits and reaches an AUC of about 0.91. That work ablates components to improve a static prediction score. It never asks whether a substantive claim about polarization depends on the BERT encoder, and it does not study temporal dynamics.
FRISS (Framing Unpacked: A Semi-Supervised Interpretable Multi-View Model of Media Frames) breaks sentences into predicate, agent, and patient views and improves frame classification on the Media Frames Corpus. We use it as the strongest measurement module, not as an object of study.
The three pipelines we audit supply the findings under test:
- the Izvestia distraction study, which pairs Granger causality with frame lexicons projected through word embeddings (Framing and Agenda-Setting in Russian News: a Computational Analysis of Intricate Political Strategies);
- the congressional study combining topic models with lagged regressions (A Frame of Mind: Using Statistical Models for Detection of Framing and Agenda Setting Campaigns);
- the Supreme Court cross-platform study.
None of the three reports a component ablation. One paper constrains our design. AblationBench finds that frontier models recover only about 45% of gold ablations and handle "replace this module" ablations much worse than "remove this module" ablations (AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research). For that reason we design our substitutes by hand rather than delegating the work to an agent.
Research Program
We build a reusable audit harness that represents each pipeline as a sequence of swappable modules: event grouping, frame or topic measurement, affect scoring, aggregation window, and the temporal model. Each module gets three substitutes.
Simple substitute. Frame measurement becomes a tuned TF-IDF logistic regression or the seed lexicons of the Media Frames Corpus (The Media Frames Corpus: Annotations of Frames Across Issues). The lagged model becomes a contemporaneous correlation, and the embedding expansion step is dropped.
Blindfolded substitute. This is a classifier trained on the same labels but given only content-free features. For text, these are platform identity, length, punctuation, capitalization, toxicity score, and named-entity masks. For time series, the blindfold keeps calendar and volume features and discards content.
Placebo substitute. Real inputs are replaced with shuffled or irrelevant ones: randomized frame labels, placebo target countries, or lags drawn under permutation.
We then recompute each paper's headline effect under every single-module swap. We also test pairs of swaps, but only for the module pairs that the single-swap results flag. This keeps the experiment count small enough to avoid testing every combination.
The second ingredient is a measurement-error correction. We hand-annotate a stratified sample of roughly 500 Reddit comments and 500 headlines from the Supreme Court corpus. From these we estimate each classifier's confusion matrix by platform. We use the matrices both to correct the observed frame distributions and to simulate matched noise. This lets us separate "Reddit is more diverse" from "our Reddit labels are noisier."
Baselines
The naive reference point is the original pipeline run as published, with its reported numbers. These are the inertia values of 1.15 and 0.71 with F1 of about 0.75 and 0.53 (Understanding Divergent Framing of the Supreme Court Controversies: Social Media vs. News Outlets); the GDP correlation of r = −0.83 and the 30-day Granger lag (Framing and Agenda-Setting in Russian News: a Computational Analysis of Intricate Political Strategies); and the 103 significant cross-party correlations (A Frame of Mind: Using Statistical Models for Detection of Framing and Agenda Setting Campaigns). Our audit counts as informative only if it changes at least one of these conclusions or certifies it against substitutes that could plausibly have overturned it.
The strongest measurement competitor is FRISS. It reports 69.7% accuracy and 60.5 Macro-F1 on the immigration subset of the Media Frames Corpus, against 67.1% for fine-tuned RoBERTa (Framing Unpacked: A Semi-Supervised Interpretable Multi-View Model of Media Frames). It matters because one could argue that any confound we find disappears once measurement is good enough. We retrain FRISS, RoBERTa, and the TF-IDF substitute on the same training data, with the same hyperparameter search budget and the same unlabeled data where the method uses it. We then rerun every downstream analysis with each of them. For cross-platform framing we found no reported FRISS result on Lacuna, so its Reddit performance will be measured, not assumed.
The strongest analysis-level competitor is a standard robustness check: varying lag length and aggregation window while keeping the measurement fixed. Our harness beats it if the blindfold or noise-matching substitutes reveal artifacts that lag and window sweeps leave invisible.
The ablation that isolates our key ingredient runs the audit twice. The first run uses only the simple and placebo substitutes. The second adds the blindfolded measurement and the noise matching. If the second run changes no verdict, the blindfold idea adds nothing and we will say so.
Why not just…?
Why not just run a multiverse or specification-curve analysis over analyst choices? This is the most serious alternative, and our simple-substitute arm is essentially one. The difference is that a multiverse varies choices made by reasonable analysts, and every measurement in it is assumed to be valid. The blindfold and matched-noise substitutes are deliberately invalid measurements, built to test whether an invalid instrument reproduces the effect. No amount of varying valid specifications reveals that a platform gap tracks classifier error. The ablation in the Baselines section settles whether this distinction matters in practice.
Why not just improve the Reddit classifier until the F1 gap closes? Better accuracy helps, but it does not answer the question. The issue is whether the errors that remain are correlated with platform, and the downstream conclusion depends on that correlation, not on average F1. The 0.53 versus 0.75 gap shows that the errors are currently far from balanced. Rerunning the analysis with FRISS, alongside confusion-matrix correction, shows directly whether better measurement alone removes the confound.
Why not just use keyword dictionaries throughout, as political science long did? That is one of our substitutes, not a rival to the audit. If lexicon-based measurement reproduces every conclusion, it is the cheaper instrument, and showing this would be a useful result in itself, in the spirit of the time-series ablations.
Why not let an LLM research agent design the ablations? AblationBench reports an F1 of 0.42 for the best model against 0.65 for PhD students, with particular weakness on "replace this module" ablations, which are exactly what our substitutes are (AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research). We will compare an agent-generated audit plan with ours as a secondary measurement only.
Decisive Experiments
The first and decisive experiment uses the Supreme Court corpus of 4,784 headlines and 41,143 Reddit comments. We add matched label noise to the news predictions and recompute the inertia gap. We also compute $R$ for a blindfolded classifier that sees platform, length, punctuation, and toxicity. If the noise-matched gap falls well below 0.44, or $R$ exceeds about 0.5, the divergence claim is substantially a measurement artifact. If the gap persists after confusion-matrix correction, the claim is strengthened against a specific threat that has not been tested before.
The second experiment applies to Izvestia, if the corpus can be obtained. We rerun the Granger test with the embedding expansion removed. We also replace the United States with placebo countries of similar coverage volume, and sweep lags from daily to quarterly. The distraction claim survives only if it is specific to the United States and holds across a band of lags rather than at 30 days alone.
The third experiment uses the congressional corpus. We vary the greedy rule that assigns 2 to 6 topics per document, and we swap the topic model for a TF-IDF clustering. We then report how many of the 103 cross-party correlations remain, separately for zero-lag and lagged correlations. Zero-lag correlations are the ones most likely to reflect shared external shocks rather than genuine responsiveness between parties.
Across all three experiments, the output is a per-finding verdict: robust, fragile to a named module, or reproduced by a blindfold. Each verdict comes with the annotated samples and the harness, so later studies can run the same audit before publishing.
References
- Blindfold Baselines for Embodied QA
- Are Language Models Actually Useful for Time Series Forecasting?
- Understanding Divergent Framing of the Supreme Court Controversies: Social Media vs. News Outlets
- Troubling Trends in Machine Learning Scholarship
- Modeling Ideological Salience and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity
- Framing Unpacked: A Semi-Supervised Interpretable Multi-View Model of Media Frames
- Framing and Agenda-Setting in Russian News: a Computational Analysis of Intricate Political Strategies
- A Frame of Mind: Using Statistical Models for Detection of Framing and Agenda Setting Campaigns
- AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research
- The Media Frames Corpus: Annotations of Frames Across Issues