Abstract
Large language models (LLMs) are becoming increasingly capable, but the mechanisms of their thinking and decision-making process remain unclear. Chain-of-thoughts (CoTs) have been commonly utilized to monitor LLMs, but this strategy fails to accurately reflect LLMs' thinking process. Techniques based on LLMs' hidden representations provide an inner perspective to monitor their latent thinking. However, previous methods only try to develop external monitors instead of making LLMs themselves easier to monitor. In this paper, we propose a novel method TELLME, improving the transparency of LLMs and helping monitors identify unsuitable and sensitive behaviors. Furthermore, we showcase the applications of TELLME on trustworthiness tasks (e.g., safety risks monitoring tasks and detoxification tasks), where LLMs achieve consistent improvement in transparency and task performance. More crucially, we theoretically analyze the improvement of TELLME on LLMs' generalization ability through optimal transport theory.
Introduction
Large Language Models (LLMs) are notoriously opaque "black boxes." Ensuring their safety and reliability typically relies on two strategies: Chain-of-Thought (CoT) monitoring and external activation monitors. However, CoT can suffer from "alignment faking," where a model provides a polite explanation while performing biased calculations. External monitors, on the other hand, often struggle with raw activations that are messy and entangled, making reliable classification difficult.
Rather than building more complex external tools, this work proposes making the model's internal representations inherently more transparent. By forcing the model to organize its latent space into distinct, manageable clusters, we can improve interpretability and monitoring accuracy.
Methodology: The TELLME Framework
We introduce TELLME (Transparency Enhancement of LLMs without External modules), a post-training refinement technique designed to reorganize an LLM’s internal representation space.
Disentanglement via Contrastive Learning
The core of TELLME is an InfoNCE loss function. During training, the model is encouraged to make internal representations of similar behaviors (e.g., two math problems) more similar, while pushing representations of different behaviors (e.g., math vs. safety violations) further apart.
Performance Maintenance
To ensure the model retains its general capabilities, we employ a dual-part retain loss:
- $L_2$ Norm Constraint: Keeps new representations close to the original ones for general tasks.
- KL Divergence: Ensures the output probability distribution remains stable compared to the original model.
Experimental Setup
TELLME was evaluated across three primary domains:
- Mathematics: MATH dataset (7 branches).
- General Knowledge: MMLU (7 domains).
- Safety: BeaverTails (5 safety risks and 1 safe category).
- General Capability: UltraChat was used to preserve conversational fluency, with evaluations on SaladBench and XSTest.
Key Results and Discussion
Testing on Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B revealed significant improvements in transparency and safety:
- Enhanced Monitoring: Representation-based monitors saw an accuracy boost of 6.3% to 6.5% due to more separable features.
- Improved Safety: In detoxification tasks, TELLME achieved a 7.5% improvement in safety rates. Simply separating "harmful" from "safe" thoughts in the latent space improved behavioral outcomes even without explicit refusal training.
- Preserved Intelligence: Performance on benchmarks like GSM8K and MMLU remained stable, with an average gap of less than 0.5%.
Visual analysis via t-SNE plots confirms that while original representations overlap significantly, TELLME-enhanced models exhibit clear, distinct clusters for different topics and safety categories.
Conclusion
TELLME demonstrates that transparency is a trainable property. By explicitly optimizing for disentangled representations, we can create models that are not only safer and more robust but also significantly easier for humans and external systems to monitor. Future work will explore integrating these transparency objectives directly into the pre-training phase to foster inherently interpretable "super-intelligent" systems.
Summary updated