The Transparency Trust Paradox

16 concepts10 papers1 proposal

A foundational assumption in explainable AI (XAI) and data engineering is that transparency breeds trust. The logic seems intuitive: if a system reveals its inner workings, performance metrics, or decision pathways, users can verify its correctness and feel more confident in its outputs.

However, empirical evidence across domains reveals a counter-intuitive phenomenon known as the transparency-trust paradox. Instead of smoothly calibrating human trust, opening the algorithmic "black box" frequently destabilizes it. Transparency does not act as a passive window; it is a cognitive intervention that can actively erode user confidence or induce dangerous overreliance.

The Mechanism of Hyper-Scrutiny

Why does seeing how a system works make us trust it less? The primary driver is that transparency shifts the user's role from a passive recipient to a critical auditor.

When a data pipeline or model exposes its internal logic, it inevitably reveals the system's underlying complexities, edge-case limitations, and imperfections. As highlighted by Sarah Oppold et al. in Trust in data engineering: reflection, framework, and evaluation methodology, providing detailed information about internal processes often triggers heightened critical scrutiny. Instead of reassuring users, this exposure can make the system appear fragile or overly convoluted.

This erosion is particularly acute among domain experts. For instance, Volkmar Pipek et al. observed in End-user interactions with intelligent and autonomous systems that exposing the underlying reasoning of autonomous systems to experienced users can lead to disillusionment, as it makes inherent algorithmic flaws and limitations glaringly obvious. Similarly, the authors of Effects of Algorithmic Decision-Making and Interpretability on Human Behavior: Experiments using Crowdsourcing demonstrate that when an algorithm reveals counter-intuitive features or unfamiliar logic, users default to skepticism rather than cooperation.

The Friction of Inconsistency

Trust is heavily dependent on predictability. When an AI system provides an explanation, that explanation must align perfectly with both the final output and the user's mental model. When it doesn't, the explanation itself becomes a source of doubt.

In a simulated clinical study by Lotte J S Ewals et al. titled The Impact of Expectation Management and Model Transparency on Radiologists’ Trust and Utilization of AI Recommendations for Lung Nodule Assessment on Computed Tomography: Simulated Use Study, radiologists showed decreased trust when they observed inconsistencies between an AI's recommendations and its generated explanations. When the system's internal reasoning appears erratic or fails to match human domain patterns, transparency transforms the system from an objective assistant into an unreliable narrator. This is echoed by Devleena Das et al. in Explainable Activity Recognition for Smart Home Systems, who found that poorly constructed or misaligned explanations paradoxically undermine confidence in otherwise accurate systems.

Overreliance vs. Bias Justification

Even when transparency does not destroy trust, it can distort human decision-making in two opposing ways: fostering a false sense of security (overtrust) or weaponizing explanations to reject valid advice (undertrust).

On one hand, explicit transparency about performance metrics can impair human vigilance. Melina A. Kunar et al. demonstrated this in Increasing transparency of computer-aided detection impairs decision-making in visual search, finding that explicitly informing users of a computer-aided detection system's accuracy metrics led to higher false alarm rates and lower overall sensitivity. The explicit metric acts as a psychological crutch, triggering overreliance and causing users to drop their guard. As Maria De-Arteaga et al. note in A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores, the risk that persuasive but flawed logic makes an erroneous output look secure remains a critical challenge for high-stakes human-in-the-loop systems.

On the other hand, transparency can empower users to ignore optimal recommendations. In economic bargaining experiments, Alexander Erlei et al. found in Impact of Algorithmic Decision Making on Human Behavior: Evidence from Ultimatum Bargaining that increasing the interpretability of a decision support system led users to query the system more but follow its advice less strictly. Instead of correcting human errors, the revealed logic allowed users to find minor flaws in the machine's reasoning, which they used to justify their own pre-existing biases.

Contextual and Structural Boundaries

The impact of transparency also depends heavily on the underlying architecture of the system and the user's environment:

  • Probabilistic vs. Deterministic Logic: As explored by Ziyan Lin et al. in Fostering Epistemic Insights into AI Ethics through a Constructionist Pedagogy: An Interdisciplinary Approach to AI Literacy, users naturally find rule-based systems more intuitive because their transparency maps directly to cause-and-effect logic. Modern machine learning models, however, are inherently probabilistic; explaining them often exposes a statistical uncertainty that alienates users when errors occur.
  • Psychological Immersion: In certain domains, transparency is actively detrimental to the core user experience. For example, David Melhart et al. note in The Ethics of AI in Games that in interactive environments like video games, exposing the underlying decision-making logic breaks immersion and disrupts psychological engagement, creating a direct conflict between compliance auditing and user experience.

The transparency-trust paradox forces a re-evaluation of how we design explainable systems. Transparency is not a universal good that automatically yields better human-AI collaboration. To build truly calibrated trust, systems must move beyond simply dumping internal data or metrics onto the user. Instead, explanations must be carefully designed to match human cognitive limits, manage expectations, and support critical thinking without triggering either hyper-skepticism or blind compliance.

Key Foundational Work

Effects of Algorithmic Decision-Making and Interpretability on Human Behavior: Experiments using Crowdsourcing (2018) — contributed 3 concept elements to this direction.
End-user interactions with intelligent and autonomous systems (2012) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_explainability_trust_paradox_e6eccfc0,
  title = {explainability trust paradox},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explainability-trust-paradox/rdr_e6eccfc05e384877b092c8c9c2b7d3d8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_explainability_trust_paradox_e6eccfc0_v1,
  title = {explainability trust paradox},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explainability-trust-paradox/rdr_e6eccfc05e384877b092c8c9c2b7d3d8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake