Computational Assessment of the Impact of Social Justice Documentaries

Jana Diesner, Susie Pak, Jinseok Kim, Kiumars Soltani, Amirhossein AleyaseniConference 2014 · DOI

Abstract: Documentaries are meant to tell a story, that is, to create memory, imagination and sharing. Moreover, documentaries aim to lead to change in people's knowledge and/or behavior. How can we know if a documentary has achieved these goals? We report on a research project where we have been developing, applying and evaluating a theoretically-grounded, empirical and computational solution for assessing the impact of social justice documentaries in a scalable, robust and rigorous fashion. We leverage cutting-edge methods from socio-technical data analytics—namely natural language processing and network analysis—for this purpose and provide a publicly available technology (ConText) that supports these routines. In this paper, we focus on the theoretical foundations of this project, address our methodological and technical framework, and provide an illustrative example of the introduced solution.

The "Missing Middle" in Impact Assessment

Measuring the "impact" of a social justice documentary is notoriously difficult. Funding agencies like the Ford Foundation or Sundance want to know if their investments are actually changing society, but current methods are often stuck in two extremes. On one hand, macro-level quantitative metrics like view counts or DVD sales show reach but lack depth. On the other, micro-level qualitative methods like focus groups provide deep insight but are expensive and impossible to scale across different films or time periods.

The authors identify a "missing middle": the ability to map the complex interplay between social agents (stakeholders, audiences) and the information they share. They argue that we need to understand not just who is talking, but what they are saying and how those conversations evolve over time.

The CoMTI Framework

To address this, the researchers propose the CoMTI (Content, Medium, Target, Impact) framework. Unlike previous linear models of communication, CoMTI incorporates feedback loops, recognizing that audiences aren't just passive recipients—they react and reshape the message.

The CoMTI framework cycle illustrating the feedback loops between Content, Medium, Target, and Impact.

As shown in the cycle above, the process is iterative. The content produced influences the medium and target, which in turn generates an impact that feeds back into the creation of future content or the refinement of the current campaign.

Operationalizing Impact with ConText

To move from theory to practice, the authors developed ConText, a publicly available socio-technical data analytics tool. The methodology involves three key models:

  1. Baseline Model: Mapping the public discourse and key players related to a film's theme before its release.
  2. Ground Truth Model: Analyzing the film's transcript to define the actual message the filmmakers intended to communicate.
  3. Model of Reality: Mapping the discourse during and after release to see if the "needle moved."

The framework is operationalized through a detailed set of dimensions and indicators, as detailed below:

Detailed breakdown of the CoMTI framework dimensions, levels, and analytics used for impact assessment.

Technically, the approach leverages Natural Language Processing (NLP) for topic modeling and sentiment analysis, and Social Network Analysis (SNA) to construct "socio-semantic networks." These networks link social agents to specific concepts, allowing researchers to see which stakeholders are championing which ideas.

Case Study: The House I Live In

The paper demonstrates its approach using a case study of the documentary The House I Live In (2012), which focuses on the "War on Drugs" and mandatory minimum sentencing (MMS). By analyzing news coverage, Twitter followers, and Facebook comments against the film's transcript, several key insights emerged:

  • The "Art vs. Message" Gap: Media coverage often centered on the director or the "event" of the screening rather than the social issue itself. While the film was successful as an "artistic event," the press coverage largely failed to pick up the specific policy arguments (MMS) the filmmakers prioritized.
  • Platform Asymmetry: Twitter was better for identifying "opinion leaders," while Facebook comments revealed how the public took the film's message in new directions, such as shifting the focus from drug laws to systemic racism.
  • Community Evolution: On Facebook, the researchers observed the public "taking the ball and running with it," moving the conversation into deeper territory that went beyond the original posts made by the film's campaign team.

Conclusion and Future Directions

While the CoMTI framework and ConText tool provide a robust foundation for scalable impact assessment, the authors acknowledge several areas for future work. Currently, the "ground truth" model relies on text-only scripts, ignoring the emotional impact of visuals and sound. Furthermore, while the study tracks public awareness and discourse, it does not yet measure "hard" impacts like legislative changes. Future efforts will focus on improving entity extraction and conducting longitudinal studies to see if shifts in discourse persist years after a film's release.

Summary updated

Abstract

Documentaries are meant to tell a story, that is, to create memory, imagination and sharing Moreover, documentaries aim to lead to change in people's knowledge and/ or behavior How can we know if a documentary has achieved these goals? We report on a research project where we have been developing, applying and evaluating a theoretically-grounded, empirical and computational solution for assessing the impact of social justice documentaries in a scalable, robust and rigorous fashion. We leverage cutting-edge methods from socio-technical data analytics -namely natural language processing and network analysis -for this purpose and provide a publicly available technology (ConText) that supports these routines. In this paper, we focus on the theoretical foundations of this project, address our methodological and technical framework, and provide an illustrative example of the introduced solution.
BibTeX
@inproceedings{diesner2014computational,
  author = {Jana Diesner and Susie Pak and Jinseok Kim and Kiumars Soltani and Amirhossein Aleyasen},
  title = {{Computational Assessment of the Impact of Social Justice Documentaries}},
  booktitle = {iConference},
  year = {2014},
  doi = {10.9776/14125},
}
Figure