TL;DR
Hidden Schema Networks (HSN) introduce a way to enforce explicit relational structures—essentially a "latent graph"—onto the output of pretrained language models like BERT and GPT-2. By encoding sentences as paths (random walks) on this graph, the model achieves better compositionality, interpretable semantic clustering, and superior performance on commonsense reasoning tasks compared to standard latent variable models.
What Problem Does This Paper Address?
While Large Pretrained Language Models (LPLMs) like GPT-4 are incredibly powerful, their knowledge is "implicit." They represent information as high-dimensional continuous vectors, which makes it difficult to understand how they compose ideas or reason through complex relationships.
Humans, by contrast, often use "schemata"—mental frameworks or relational structures—to organize knowledge. We understand a "dog" not just as a cluster of pixels or a word frequency, but through its relations to "animal," "barking," and "pet." The authors aim to bridge this gap by forcing LPLMs to map natural language onto a discrete, relational latent space (a graph), enabling the model to perform "combinatorial generalization"—the ability to compose known units into novel, complex ideas.
Methods
The authors propose the Hidden Schema Network (HSN), which operates within a Variational Autoencoder (VAE) framework:
- The Latent Graph: The model maintains a global latent graph where nodes represent abstract "symbols" or concepts.
- The Encoder (BERT-based): A sentence is processed by a pretrained BERT model. A Transformer-based mapping then translates the BERT output into a sequence of symbols. Crucially, this sequence is modeled as a biased random walk across the latent graph.
- The Decoder (GPT-based): To generate text, the model uses a modified GPT-2. It employs a Pseudo-Self Attention (PSA) mechanism, which allows the decoder to "attend" to the inferred schema (the path on the graph) without overwriting its own pretrained weights.
- Inductive Bias: Unlike standard VAEs that use unstructured Gaussian noise, HSN uses the topology of the graph as an inductive bias. This forces the model to learn that certain concepts (nodes) are connected and others are not.
Datasets
The researchers evaluated HSN across three distinct environments:
- Synthetic Data: Random token sequences generated from ground-truth graphs (Barabási-Albert and Erdös-Rényi topologies) to see if the model could "uncover" a hidden structure.
- Natural Language Corpora: The Penn Treebank (PTB), Yahoo, and Yelp datasets were used to test if the inferred symbols captured real-world semantics like topics or sentiments.
- Commonsense Reasoning: The ATOMIC dataset, which contains "If-Then" knowledge triples (e.g., "If X buys a gift, then X wanted to be nice"), was used to test the model's ability to generate logical conclusions.
Key Insights and Ideas
- Small-World Semantics: The graphs inferred from natural language naturally developed "small-world" properties—high clustering and short path lengths. This mirrors how human semantic memory is thought to be organized.
- Topological Clustering: When visualizing the latent graph, the authors found that different categories (e.g., "Science" vs. "Sports" in Yahoo) occupied distinct, well-connected regions of the graph. "Science" schemata were closer to "Education" than to "Sports," showing the graph captured genuine semantic proximity.
- Reasoning as Path Completion: The authors framed reasoning as a sequence-to-sequence problem on the graph. By learning to predict the "second half" of a random walk (the "Then" part) given the "first half" (the "If" part), the model could generate more coherent commonsense inferences.
- Efficiency of Discrete Symbols: HSN outperformed dense-vector baselines (like iVAE and Optimus) in both Perplexity and Mutual Information, suggesting that discrete, structured representations might be more efficient for language modeling than purely continuous ones.
Key Results
- Graph Recovery: On synthetic data, HSN achieved a ROC AUC of up to 0.989, proving it can successfully reconstruct hidden relational structures from raw sequences.
- Language Modeling: On the Yelp dataset, HSN achieved a Perplexity (PPL) of 19.72, significantly better than the fine-tuned GPT-2 baseline (23.40) and other VAE-based models.
- Commonsense Performance: On the ATOMIC reasoning task, HSN's "reasoning" paths helped it achieve a BERT Score of 0.694, nearly doubling the performance of a standard COMET (GPT-2) model (0.385) and even outperforming much larger models like GPT-2-XL.
Limitations and Future Work
- Decoder Fine-tuning: While the BERT encoder can remain mostly frozen, the GPT decoder requires fine-tuning to learn how to interpret the new symbolic schemata. This limits the "plug-and-play" utility with massive, closed-source models.
- Sensitivity to Path Errors: In the reasoning tasks, the authors noted that if the model misses even a single symbol in a random walk, the "reasoning" can go off the rails. They suggest exploring more robust encoding schemes.
- Scaling Up: The authors propose using Parameter-Efficient Fine-Tuning (like LoRA) in the future to apply HSN to even larger "not-so-large" decoder models without massive computational overhead.
Summary updated