Dimensions of Transparency in NLP Applications

Michael Saxon, Sharon Levy, Xinyi Wang, Alon Albalak, William Yang Wang

Abstract

Broader disclosive transparency$-$truth and clarity in communication regarding the function of AI systems$-$is widely considered desirable. Unfortunately, it is a nebulous concept, difficult to both define and quantify. This is problematic, as previous work has demonstrated possible trade-offs and negative consequences to disclosive transparency, such as a confusion effect, where "too much information" clouds a reader's understanding of what a system description means. Disclosive transparency's subjective nature has rendered deep study into these problems and their remedies difficult. To improve this state of affairs, We introduce neural language model-based probabilistic metrics to directly model disclosive transparency, and demonstrate that they correlate with user and expert opinions of system transparency, making them a valid objective proxy. Finally, we demonstrate the use of these metrics in a pilot study quantifying the relationships between transparency, confusion, and user perceptions in a corpus of real NLP system descriptions.

Introduction

While the call for "transparent" AI is universal, defining and measuring it remains a significant challenge. Regulatory frameworks, such as those proposed by the EU, emphasize the need for AI capabilities to be communicated in an "appropriate manner." However, without objective benchmarks, developers often struggle with disclosive transparency: the extent to which detailed messaging about a system is provided to stakeholders. A critical hurdle is the "confusion effect," where an overabundance of technical detail can paradoxically diminish user trust and comprehension. This research aims to bridge this gap by developing objective, neural language model-based metrics to quantify transparency.

Methodology

The proposed framework decomposes transparency into two primary dimensions: Replicability and Style-Appropriateness.

Replicability

Replicability assesses whether a description contains sufficient information to theoretically reconstruct the system. This is modeled through two proxies:

  1. Trigram Information Recovery ($R_R$): A GPT-2 model is tasked with expanding a brief abstract into a detailed description. The metric calculates the proportion of original trigrams (three-word sequences) recovered by the model.
  2. Sentence Affinity ($R_A$): Utilizing BERTScore, the model evaluates how effectively each sentence in a brief description covers the information present in the full technical documentation.

Style-Appropriateness

Style-Appropriateness ($C$) measures whether the text's technical level is suited for its intended audience. This is calculated using a log-likelihood ratio between two models: one fine-tuned on a technical corpus (arXiv) and a "vanilla" model trained on general web text. This allows for the classification of text as either "academic" or "public-facing."

Overview of the proposed transparency metrics framework, illustrating the derivation of Style-appropriateness and Replicability scores.

Experimental Setup

The models were trained on a substantial corpus of 130,000 LaTeX files from arXiv (categories cs.* and cs.CL) from 2007 to 2020. For evaluation, the researchers utilized the ACL System Demo Corpus, comprising 268 abstracts from major NLP conferences. A subset of 55 abstracts describing systems for laypeople (e.g., translation tools) was used for the user study.

Results and Discussion

The automated metrics demonstrated significant alignment with human judgment. Specifically, the Sentence Affinity metric showed a positive correlation with expert ratings of function transparency ($PCC = 0.276$).

Correlation analysis showing the relationship between automated replicability metrics (Trigram Recovery and Affinity Rate) and mean human transparency ratings.

The Illusion of Understanding

A key finding was the distinction between Task Transparency (what a system does) and Function Transparency (how it works). Users often reported a high perceived understanding of a system's function simply because the task was explained clearly, even if the underlying architecture remained opaque. This "illusion of understanding" suggests that clarity in task description can inflate user confidence.

Style as a Driver of Trust

The Style-Appropriateness metric proved to be a stronger predictor of user understanding than raw information density. This highlights that the "vibe" or stylistic alignment of the writing is as crucial for perceived transparency as the factual content itself.

Conclusion and Future Work

This study provides a foundation for objective transparency measurement, though it is limited by a small pilot sample size and a focus on academic abstracts. Future research should extend these metrics to consumer-facing documents like Terms of Service. Ultimately, these metrics could enable Natural Language Generation systems to automatically refine technical documentation into more transparent, audience-appropriate formats.

Summary updated

Abstract

Broader disclosive transparency$-$truth and clarity in communication regarding the function of AI systems$-$is widely considered desirable. Unfortunately, it is a nebulous concept, difficult to both define and quantify. This is problematic, as previous work has demonstrated possible trade-offs and negative consequences to disclosive transparency, such as a confusion effect, where "too much information" clouds a reader's understanding of what a system description means. Disclosive transparency's subjective nature has rendered deep study into these problems and their remedies difficult. To improve this state of affairs, We introduce neural language model-based probabilistic metrics to directly model disclosive transparency, and demonstrate that they correlate with user and expert opinions of system transparency, making them a valid objective proxy. Finally, we demonstrate the use of these metrics in a pilot study quantifying the relationships between transparency, confusion, and user perceptions in a corpus of real NLP system descriptions.
BibTeX
@inproceedings{saxon2021dimensions,
  author = {Michael Saxon and Sharon Levy and Xinyi Wang and Alon Albalak and William Yang Wang},
  title = {{Dimensions of Transparency in NLP Applications}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=dnIPfPN317T},
}
Figure