Reliable AI Needs to Externalize Implicit Knowledge: A Human-AI Collaboration Perspective

Hengyu Liu, Tianyi Li, Zhihong Cui, Yushuai Li, Zhangkai Wu, Torben Bach Pedersen, Kristian Torp, Christian S. JensenarXiv 2026 · arXiv

TL;DR

Current AI reliability methods focus heavily on verifying explicit facts, leaving the underlying reasoning patterns and intuitions (implicit knowledge) completely unverified. This position paper introduces Knowledge Objects (KOs)—structured, human-readable artifacts designed to externalize an AI's implicit reasoning so humans can inspect, verify, and accumulate validation over time. By shifting the economics of verification, KOs turn one-off human corrections into permanent, compounding organizational reliability.

What Problem Does This Paper Address?

AI capability is skyrocketing, but reliability is lagging behind. Even when equipped with advanced techniques like Retrieval-Augmented Generation (RAG), specialized legal AIs still hallucinate 17–34% of the time, and LLMs frequently exhibit extreme overconfidence in incorrect answers.

The authors argue that this stems from a fundamental verification gap. AI models learn from two types of knowledge:

  1. Explicit knowledge: Documented, citable facts (e.g., textbooks, databases) which make up only 5–20% of organizational knowledge.
  2. Implicit knowledge: Undocumented reasoning patterns, problem-solving strategies, and domain intuitions (80–95% of knowledge) embedded silently within training data conversations and logs.
Figure 1. AI training data spans data, information, and knowledge. Within knowledge, only the explicit fraction (5–20%) is docu- mented and verifiable; the implicit majority (80–95%) consists of undocumented patterns that drive capability but resist verification.

Existing reliability frameworks are entirely AI-centric. RAG checks what sources say, but not how the AI reasons about them. Alignment techniques like RLHF bury knowledge invisibly inside billions of model parameters. Because the AI's implicit reasoning remains invisible and untraceable, humans cannot effectively audit or correct the core logic driving the model's most valuable capabilities.

Methods

Instead of proposing a new model architecture or training algorithm, the paper introduces a conceptual and architectural framework centered on Knowledge Objects (KOs).

A KO is not a specific data type, but a property profile that any externalized piece of knowledge must satisfy. It consists of four core components: a knowledge claim/procedure, supporting evidence, explicit scope/limitations, and validation metadata. To bridge the verification gap, a KO must fulfill five essential properties:

  • Understandable: Expressed in human-readable text (claims, evidence, scope) rather than opaque vector embeddings.
  • Verifiable: Contains explicit metadata tracking who validated the claim, when, and with what confidence level.
  • Traceable: Maintains clear provenance of where the claim originated and how it has evolved.
  • Controllable: Allows humans to directly modify, annotate, or reject specific parts of the knowledge.
  • Reusable: Persists across sessions and users so that validation efforts are not lost.

The authors outline the KO-hub collaboration paradigm, where the AI automatically extracts candidate KOs from human-AI interaction data. Human experts then audit and sign off on these objects, which accumulate into a shared repository that the AI can safely retrieve in future tasks.

Figure 2. The KO-hub collaboration paradigm.

Datasets

Not discussed in the paper. As a position paper, it focuses on establishing a theoretical framework and research roadmap rather than introducing a new dataset or empirical benchmark.

Key Insights and Ideas

  • Flipping Verification Economics: Historically, humans didn't document implicit knowledge because the cost of writing it down exceeded its perceived value. KOs change this dynamic: by having the AI automatically structure its implicit reasoning into inspectable drafts, the human's role shifts from creation to validation, drastically lowering the economic barrier to capturing expert intuition.
  • Amortizing Human Effort: Currently, AI safety is inefficient because every user must independently evaluate whether an AI's output is trustworthy. KOs allow human validation to compound; once a senior engineer verifies a specific reasoning pattern, that trust is recorded and inherited by every subsequent user.
  • Understandable $\neq$ Verifiable: The authors point out a common misconception in AI explainability: Chain-of-Thought (CoT) prompting makes an AI's step-by-step process understandable, but it does not make it verifiable. CoT explanations can be unfaithful (not reflecting the true internal logic) and lack any mechanism to persist human judgment.

Key Results

As a position paper, the primary contribution is qualitative. However, the authors contextualize their proposal by synthesizing critical failure modes of current state-of-the-art methods:

Table 1. Current methods and their effect on implicit knowledge.

The paper highlights that current self-correction and internal verification methods fail because they merely measure internal model consistency rather than correspondence with objective truth. For instance, verbalized confidence intervals are severely miscalibrated, with nominal 99% confidence intervals mapping to actual correct answers only 65% of the time.

Limitations and Future Work

The authors acknowledge several open challenges that form their future research agenda:

  • Avoiding User Friction: If creating or validating KOs requires tedious form-filling, human adoption will plummet. The backend infrastructure must automatically extract KOs, keeping human interaction as lightweight as a single click or a natural annotation.
  • Governance and Conflicts: In large organizations, establishing who has the authority to validate specific KOs and resolving conflicting opinions between human validators remains an open socio-technical challenge.

Roadmap: The authors propose a multi-horizon timeline, starting with near-term (1–2 years) single-domain pilots that add KO properties to existing agent frameworks (like Voyager or ExpeL), moving toward cross-domain schemas (2–4 years), and culminating in community-wide, Wikipedia-style KO ecosystems (4+ years).

Summary updated

Abstract

This position paper argues that reliable AI requires infrastructure for human validation of implicit knowledge. AI learns from both explicit knowledge (papers, documentation, structured databases) and implicit knowledge (reasoning patterns, debugging processes, intermediate steps). Implicit knowledge remains unexternalized because documentation cost exceeds perceived value -- yet AI learns from it indiscriminately, acquiring both beneficial patterns and harmful biases. Current reliability methods can only verify explicit knowledge against sources, creating a fundamental gap: the most valuable AI capabilities (reasoning, judgment, intuition) are precisely those we cannot verify. We propose Knowledge Objects (KOs) -- structured artifacts that externalize implicit knowledge into forms humans can inspect, verify, and endorse. KOs transform verification economics: what was previously too costly to verify becomes feasible, enabling accumulated human validation to improve reliability over time.
BibTeX
@misc{liu2026reliable,
  author = {Hengyu Liu and Tianyi Li and Zhihong Cui and Yushuai Li and Zhangkai Wu and Torben Bach Pedersen and Kristian Torp and Christian S. Jensen},
  title = {{Reliable AI Needs to Externalize Implicit Knowledge: A Human-AI Collaboration Perspective}},
  year = {2026},
  eprint = {2605.02010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.02010},
}
Figure