TL;DR
This paper introduces a computational method to quantify "politicization" in foreign aid project reports using document embeddings. By training a Doc2Vec model on a large corpus of USAID reports, they create a metric that measures how closely a report aligns with politicized keywords, offering a more nuanced approach than previous blunt operationalizations.
What Problem Does This Paper Address?
Foreign aid effectiveness is often linked to whether aid is provided for political or altruistic reasons. However, measuring "politicization" – the extent to which aid decisions are driven by political interests rather than purely developmental ones – has been challenging. Existing methods are often blunt, relying on broad categorizations or donor characteristics. Furthermore, large datasets like USAID's Development Experience Clearinghouse (DEC), containing thousands of project evaluation reports, are largely unstructured and difficult to analyze for this specific concept. This paper aims to bridge this gap by developing a computational metric for politicization using natural language processing.
Methods
The core of the methodology involves using Doc2Vec, a technique that learns dense vector representations for documents. Specifically, they employed the Distributed Memory Paragraph Vector (DMPV) variant, which considers word order, to capture richer semantic information.
- Pre-training: To overcome the limitation of a smaller in-domain dataset, they initialized their Doc2Vec model with pre-trained word embeddings from the Common Crawl corpus.
- Document Embeddings: The Doc2Vec model was trained on a corpus of USAID health project reports to generate 300-dimensional vectors for each report.
- Politicization Metric: A "politicization metric" was created by calculating the cosine similarity between the average vector of a set of manually identified "politicized" keywords and the vector of each individual project report.
- Keyword Classification: Keywords were derived from the USAID thesaurus and classified by political scientists into three categories: politicized (e.g., reproductive health terms), non-politicized (e.g., general health terms), and potentially politicized (terms that could relate to either).
Datasets
- Training Data: A corpus of 99 USAID health project evaluation reports, sampled to be representative across years and countries, totaling approximately 1.6 million tokens.
- Evaluation Data: The same 99 reports were used for evaluation, with a "silver standard" score derived from existing metadata in the DEC. This silver score was calculated by averaging the politicization scores of keywords associated with each report in the DEC's metadata.
Key Insights and Ideas
- Operationalizing Abstract Concepts: The paper demonstrates that abstract concepts like "politicization" can be computationally operationalized using document embeddings, moving beyond simple keyword matching.
- Doc2Vec's Potential: The success of Doc2Vec on related tasks (sentiment analysis, topic modeling) suggests its suitability for capturing nuanced semantic information relevant to political science concepts.
- Beyond Blunt Metrics: The study highlights the limitations of traditional, blunt operationalizations of politicization and proposes a more granular, text-based approach.
- Context Matters: The authors note that politicization can be context-dependent, and a single vector representation for an entire document might be too coarse-grained. Segmenting reports into intervention descriptions could yield better results.
- "Silver Standard" Evaluation: The use of DEC metadata as a "silver standard" for evaluation, while not ideal, provides a pragmatic way to assess the metric's validity when a true "gold standard" is infeasible.
Key Results
- The developed politicization metric showed a weak but statistically significant correlation (Spearman's ρ = 0.280, p = 0.005) with the "silver score" derived from DEC metadata.
- Reports with higher similarity scores to politicized keywords tended to cover more politicized topics, as indicated by the silver score.
- The study identified specific reports that either aligned well with the metric or served as interesting outliers, providing qualitative insights into the model's behavior. For instance, a malaria project report with a high politicization score was linked to its objective of expanding "malaria in pregnancy activities."
Limitations and Future Work
- Dependence on Manual Annotation: The current method relies on manually annotated politicized keywords, which limits generalization beyond the health sector. Future work could involve inducing keywords from external political data sources.
- Document Granularity: Representing an entire document with a single vector is considered too coarse. Future work should explore segmenting reports into descriptions of specific interventions.
- Limited Domain-Specific Resources: The authors faced limitations in hyperparameter tuning and intrinsic evaluation due to a lack of domain-specific resources.
- Conceptualization Challenges: Defining and grounding "politicization" in textual data remains a challenge.
- Static Embeddings: The use of static embeddings precludes observing how politicization changes over time or across different contexts for the same topics.
- Future Directions: The authors suggest integrating more explicitly political variables, exploring contextualized embedding-based methods, and using the metric to study the relationship between politicization and project effectiveness. They also propose developing a politicization metric for keywords at the document level.
Summary updated