Unsupervised Probing of Latent Knowledge in Language Models

67 concepts55 papers1 proposal

When a large language model (LM) is trained on billions of sentences, its primary objective is simple: predict the next token. Yet, to excel at this statistical guessing game, the model cannot merely memorize surface-level character sequences. It must build an internal representation of the world described by that text.

A major realization in modern natural language processing is that pre-trained language models act as implicit, high-dimensional knowledge bases. The factual, relational, and structural knowledge of human society is woven directly into the model's weights during unsupervised pre-training. Rather than training specialized models for every downstream task, researchers have found that we can directly "probe" or externalize this latent knowledge without any task-specific fine-tuning.

Language Models as Implicit Knowledge Bases

Why does an objective as simple as token prediction yield deep world knowledge? As Tadahiro Taniguchi proposes through the Collective Predictive Coding hypothesis, human language corpora act as a decentralized repository of collective intelligence and shared sensorimotor experiences. By optimizing for language modeling, neural networks implicitly absorb this distributed intelligence.

Even before the dominance of transformers, early distributional semantics captured structured factual information. Abhijeet Gupta et al. demonstrated that high-dimensional skip-gram vectors implicitly encode referential attributes—such as geographic coordinates or demographic data—purely as a byproduct of linguistic co-occurrence patterns.

With the advent of large-scale pre-trained transformers, this capability expanded dramatically. In their seminal work introducing the LAMA (LAnguage Model Analysis) probe, Fabio Petroni et al. showed that pre-trained models like BERT contain a wealth of relational and factual knowledge. Without any task-specific fine-tuning, these models could match or exceed traditional, supervised relation extraction systems on factual queries. As reviewed by Danai Koutra et al., this internal repository spans vast domains, allowing LMs to serve as viable, unsupervised alternatives to manually curated databases.

Standard Generation Bottleneck:
[Prompt] ---> [Transformer Layers] ---> [Vocabulary Softmax] ---> [Discrete Token]
                                                |
Unsupervised Probing (Bypassing the Bottleneck): |
                                                v
                                  [Internal Vector Geometry] 
                                  (Poincaré Distance / Similarity Maps)
                                                v
                                  [Explicit Relational Knowledge]

Bypassing the Generation Bottleneck

Extracting this latent knowledge is not always as simple as asking the model a question. Standard text generation forces the model's internal state through a vocabulary bottleneck (the final softmax layer), which can obscure sophisticated linguistic understanding. Unsupervised probing bypasses this bottleneck by looking directly at the geometry of the model's internal activations, attention maps, and hidden states.

Exploiting Internal Attention Maps

Instead of reading the model's output tokens, we can look at how its attention heads route information. Tassilo Klein et al. demonstrated that the internal attention mechanisms of an un-tuned BERT model can directly solve difficult commonsense reasoning tasks like the Winograd Schema Challenge. By introducing a "Maximum Attention Score" (MAS) to aggregate attention across layers and heads, the authors proved that the model dynamically links ambiguous pronouns to their correct referents. Further validating this, Leyang Cui et al. quantified these internal concept links, showing that BERT's attention and attribution scores align with external, human-curated knowledge graphs significantly better than chance.

Geometric and Vector-Space Probing

Another approach relies on measuring distances within the continuous embedding space. For instance, standard prompt-based generation often fails on subtle tasks like identifying whether a word shares the same meaning across two different contexts. To solve this, Mohsen Tabasi et al. introduced Similarity Prompting, a method that extracts internal vector representations and compares their contextualized embeddings directly, achieving the performance of fully supervised models without task-specific training.

When dealing with structured data, like mapping natural language queries to database schemas, the geometry of the embedding space becomes crucial. Lihan Wang et al. developed the PROTON framework, which uses Poincaré (hyperbolic) distance to probe schema-linking information. Because hierarchical structures naturally map to hyperbolic spaces, measuring distances in this projection surfaces latent semantic links that traditional string-matching techniques miss. Similarly, Aiwei Liu et al. showed that by masking specific tokens and measuring the resulting shifts in embedding space, a model can infer complex schema relationships without explicit alignment labels.

Externalizing Latent Knowledge into Explicit Structures

Once we establish that this knowledge exists within the continuous representations of the model, we can use targeted probing to externalize it into structured, human-readable formats.

  • Knowledge Graph Construction: Rather than extracting facts from text via rigid rules, Antoine Bosselut et al. introduced COMET, demonstrating that pre-trained transformers can be fine-tuned on a small set of structured examples to act as generative knowledge bases, automatically populating explicit relational tuples.
  • Structural and Text Mining: Yu Zhang et al. outlined a roadmap for weakly supervised text mining, showing how topics, entities, and taxonomies can be extracted with minimal human effort by tapping into the latent world knowledge of LMs. For network analysis, Zequn Liu demonstrated in the MetaFill framework that an LM's internal semantic similarity scores correlate directly with the actual connectivity of nodes in heterogeneous information networks, allowing for unsupervised meta-path generation.
  • Linguistic and Semantic Parsing: Pre-trained models also contain latent structural intuitions about language itself. Hoang H Nguyen et al. leveraged unsupervised probing to induce semantic slots (like dates or locations) from user utterances without token-level labels. In the domain of event extraction, Heng Ji et al. showed that prompt-based infilling tasks can surface an open vocabulary of event-specific argument roles without requiring human-defined ontologies.

This capacity extends to highly abstract or implicit concepts. Joe Davison et al. mined commonsense facts by measuring the internal coherence of sentences using Pointwise Mutual Information (PMI). In specialized domains, Girishkumar Ponkiya et al. used simple templates to probe models for noun compound interpretations, while Li Lin et al. utilized LLMs to map fragmented, implicit conceptual definitions across large scientific corpora. Even when attributes are completely omitted from text, Henry Peng Zou et al. demonstrated that models can leverage internal prior knowledge to infer implicit product properties from indirect contextual hints.

Managing the Fragility of Latent Knowledge

While the baseline knowledge inside pre-trained models is vast, it is often disorganized and fragile. Kyle Richardson et al. discovered that although models possess a surprising baseline understanding of concept definitions and hierarchies, their reasoning can quickly break down when faced with multi-hop logic or clever distractors. Furthermore, as Harish Tayyar Madabushi et al. noted, complex linguistic structures—such as constructional pairings of form and meaning—are redundantly embedded within general lexico-semantic representations rather than organized as isolated, clean modules.

To reliably exploit this latent knowledge, researchers employ two primary strategies:

  1. Elicitation via Structured Reasoning: Instead of querying the model for a direct answer, frameworks like CAGE by Nazneen Fatema Rajani et al. force the model to "think out loud." Fine-tuning the model to generate natural language explanations triggers the extraction of its internal world knowledge into structured logic that downstream components can safely use. This internal capability is robust enough that, as shown by Martin Fajcik et al., models like RoBERTa can implicitly acquire the commonsense knowledge necessary to identify and extract complex counterfactual (hypothetical) statements.
  2. Contrastive and Negative Reinforcement: Because un-tuned models can achieve high baseline accuracy on abstract concepts—often exceeding 60% zero-shot, as noted by Xin Xie—we can sharpen their boundaries by using the model's own near-miss predictions as "hard" negative examples during training. This negative augmentation forces the model to better differentiate subtle semantic boundaries.

By treating pre-trained language models not just as text generators, but as continuous, geometric knowledge graphs, unsupervised probing provides a powerful, parameter-free window into what AI models truly know about the world.

Key Foundational Work

Explain Yourself! Leveraging Language Models for Commonsense Reasoning (2019) — contributed 2 concept elements to this direction.
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction (2019) — contributed 2 concept elements to this direction.
Distributional vectors encode referential attributes (2015) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_semantic_knowledge_c30b4e89,
  title = {latent semantic knowledge},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-semantic-knowledge/rdr_c30b4e89c248476d80bbd0a9e213b41e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_semantic_knowledge_c30b4e89_v1,
  title = {latent semantic knowledge},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-semantic-knowledge/rdr_c30b4e89c248476d80bbd0a9e213b41e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake