When we interact with large foundation models, we often treat them as highly malleable systems. We craft optimal prompts, surgically alter internal activations at inference time, or apply preference optimization algorithms to shape their behavior. These techniques give the compelling illusion of teaching a model new capabilities or concepts.
However, a unified insight has emerged across recent research: post-pretraining interventions cannot create new knowledge; they can only navigate and surface the pre-existing latent space. Whether you are optimizing a prompt, steering an attention head, or fine-tuning with human preferences, your intervention is strictly bounded by the representational capacity established during the initial pre-training phase. This boundary represents a hard ceiling on model adaptation.
The Geometry of the Latent Space
To understand why this ceiling exists, we must look at how models represent information. During web-scale pre-training, a model constructs a high-dimensional manifold—a latent space—where concepts, styles, and factual associations are mapped as vectors. Every downstream technique operates by shifting, projecting, or filtering activations within this pre-established space. If a concept, factual connection, or structural rule was never encoded during pre-training, its corresponding coordinates or directions simply do not exist.
[ Pre-training Phase ] ---> Establishes the Latent Manifold (The Map)
|
v
[ Post-Pretraining ] ---> Prompting / Steering / Alignment (Navigation)
* Cannot steer to a destination not on the map *
Prompting as Latent Space Search
Prompt engineering is often viewed as a way to unlock a model's hidden potential. For instance, Minzheng Wang et al. developed PromISe, an automated framework that introspects and optimizes prompts for specific target models. While this significantly boosts benchmark scores, the authors demonstrate that these gains are ultimately constrained by the target model's architectural limits and training history. Optimization can only surface existing latent knowledge; it cannot synthesize capabilities that were never encoded.
This boundary is not just an empirical observation; it is a theoretical limit. Prashant Trivedi et al. introduced Align-Pro, a framework treating prompt optimization as a formal mathematical problem for alignment. While they prove that input optimization can match the performance of full parameter fine-tuning, their formulation reveals a fundamental catch: prompt-based alignment is strictly bounded by the inherent knowledge and reasoning capacity of the underlying frozen model. No degree of input manipulation can elicit information that is missing from the model's parameters.
Inference-Time Steering and the Limits of Activation Editing
If prompts merely search the input space, what happens when we directly alter a model's internal activations? Activation steering attempts to "dial up" specific attributes by adding vectors to hidden states.
For example, Shaolei Zhang et al. created TruthX to isolate and edit a "truthful" dimension in Large Language Models to alleviate hallucinations. However, the authors note that this internal representation editing is entirely dependent on the underlying training data. If a fact was never learned during pre-training, no amount of activation steering can force the model to produce it accurately.
This hard limit applies across modalities and tasks:
- Generative Music: Junghyun Koo et al. proposed SMITIN to steer musical attributes via classifier probes applied to attention heads. They found that if a musical concept was insufficiently represented in the training data, the internal probes failed to find any actionable signal for control.
- Multi-Attribute Control: Duy Lap Nguyen et al. showed with MAT-STEER that inference-time steering hits a hard wall when a desired behavior is completely absent from the latent space, rendering the model unsteerable for that attribute.
- Reasoning Elicitation: Even when trying to unlock complex reasoning—such as the gradient-based representation optimization proposed by Zhen Wang 0033 et al. to elicit Chain-of-Thought capabilities—the intervention remains fundamentally constrained by the foundation model's pre-existing data exposure and logic capacity.
This ceiling is equally visible in vision models. Heliang Zheng et al. discovered that perturbing the "null-text" unconditional branch in Classifier-Free Guidance acts as a hidden "cartoon-style creator." Yet, the authors emphasize that this training-free stylization is entirely bounded by the pre-existing artistic concepts residing in the base model's weights. Similarly, the C3 framework by Jiyeon Han et al. amplifies low-frequency features in the Fourier domain to boost creative generation, but it cannot synthesize structural variations or concepts that were never encountered during the initial training phase.
Why Alignment and Fine-Tuning Cannot Synthesize Knowledge
One might assume that preference optimization (like RLHF or DPO) or mechanistic editing can bypass these limits because they update model weights. However, architectural evidence suggests these methods primarily reconfigure existing associations rather than expanding the model's fundamental capacity.
Zhiqing Sun et al. demonstrated this with SELF-ALIGN, showing that while minimal human supervision can successfully align a model to human values, the procedure cannot generate factual accuracy if the underlying base model lacks the necessary information to address a query.
When aligning text-to-image and video models, advanced optimization frameworks encounter the exact same barrier:
- CaPO: Kyungmin Lee et al. developed Calibrated Multi-Preference Optimization, noting that fine-tuning cannot easily introduce complex capabilities (like accurate text rendering) if the base model lacks the underlying representational capacity.
- RealDPO: Jianlou Si et al. used real-world videos as a gold standard for preference learning, but observed that if the base architecture has not captured a specific physical law during initial training, alignment methods cannot synthesize that missing knowledge.
- PCPO: Jeongjae Lee et al. stabilized reinforcement learning for image generation, confirming that the base model's inability to generate higher-quality outputs can obscure the performance gains provided by advanced optimization strategies.
Alignment simply shifts the probability distribution toward the best available representations within the model's pre-existing capacity. It acts as a filter, not an expansion pack.
Interpretability as a Mirror of Latent Content
This dependency also dictates the boundaries of explainable AI and model editing. Post-hoc interpretability methods like AnyCBMs, introduced by Gabriele Dominici et al., map internal hidden embeddings to human-understandable concepts to create Concept Bottleneck Models. The authors explicitly note that this extraction is fundamentally constrained by the features already encoded; you cannot extract or explain using a concept that the model never internally represented.
This reality is mirrored in mechanistic editing and extractive alignment. Joykirat Singh et al. show that editing techniques function by reconfiguring internal associations rather than expanding the underlying knowledge base. Similarly, extractive alignment methods like ResiDual (Lorenzo Basile et al.), which reweight the principal components of specialized heads in the residual stream, are strictly limited to the existing feature space of the pre-trained model.
The Core Takeaway
Every post-pretraining technique—whether it operates on the input (prompting), the internal activations (steering), or the token probabilities (alignment)—is an act of navigation, not creation.
The pre-training phase defines the boundaries of the map. Interventions can guide the model to specific coordinates with remarkable efficiency, but they cannot steer it off the edge of its known world. To achieve true breakthroughs in downstream tasks, we cannot rely solely on smarter interventions; we must expand the representational capacity of the base models themselves.
Key Foundational Work
AnyCBMs: How to Turn Any Black Box into a Concept Bottleneck Model (2024) — contributed 2 concept elements to this direction.
TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space (2024) — contributed 1 concept elements to this direction.
Null-text Guidance in Diffusion Models is Secretly a Cartoon-style Creator (2023) — contributed 1 concept elements to this direction.