When we interact with Large Language Models (LLMs), we frequently use their tone, vocabulary, and structural fluency as proxies for their intelligence and reliability. If a model generates a radiology report or a legal brief using flawless professional jargon, we naturally assume the underlying facts are accurate.
However, a growing body of research reveals a troubling phenomenon: stylistic proficiency and factual accuracy are fundamentally decoupled. Forcing a model to adopt a specific persona, professional style, or human-like distribution often directly triggers a collapse in its truthfulness, yielding highly authoritative, polished errors that are exceptionally difficult for humans to detect.
The Illusion of Competence
The tension between how a model speaks and what it actually knows becomes glaringly obvious when LLMs are deployed in specialized domains. We often observe "professionalized hallucinations"—outputs that look structurally perfect but are factually vacant.
For instance, Jiang et al. demonstrated that while advanced prompting can dramatically improve the professional tone and stylistic coherence of generated radiology reports, it fails to improve underlying clinical accuracy. The model sounds like a radiologist, but its medical assessments remain incorrect.
Similarly, Liu et al. evaluated LLMs on financial data narration—the task of turning complex financial tables into analytical stories. They found that while models easily mimic the sophisticated style of a financial analyst, they struggle with basic numerical accuracy, failing to maintain factual correctness on core analytical operations. Stylistic proficiency, therefore, acts as an unreliable mask for actual task competence.
Why Fine-Tuning Mimics Style Instead of Facts
To understand why this decoupling occurs, we must look at what happens during the optimization process. When a model is fine-tuned on a specialized corpus (such as legal documents or patent applications), it faces strong optimization pressure to minimize token prediction loss.
The easiest way for a model to minimize this loss is to copy the highly predictable structural and stylistic patterns of the target domain—the vocabulary, sentence lengths, and formatting. Learning the facts or the underlying logic requires a much deeper, more complex internal reorganization. As a result, the model defaults to a superficial shortcut: it learns to "pattern copy" the style without internalizing the truth.
Ghosh et al. show that instruction tuning often acts as a superficial makeover; models prioritize matching the length and tone of training examples over factual correctness, fabricating details simply to satisfy stylistic expectations. This superficial adaptation is documented across multiple domains:
- Legal and Patent Prose: Knappich et al. observe that optimizing models for specialized patent drafting causes them to invent technical details to satisfy expected linguistic structures. In the legal domain, Marques Peres shows that domain-specific fine-tuning leads to structurally perfect but entirely fabricated legal citations.
- Knowledge Internalization Failure: Bouvard et al. demonstrate that fine-tuning on new domain data successfully teaches the model the target vocabulary, but results in confident hallucinations regarding specific details like dates and locations.
- The Paradox of Factual Training: Strikingly, Zhang et al. discover that even fine-tuning directly on factual datasets can paradoxically reduce accuracy. The model learns to replicate the confident, authoritative linguistic style of factual statements, leading it to guess plausible-sounding but incorrect information when a prompt exceeds its pre-trained knowledge boundaries.
The Optimization Tug-of-War
This tension isn't limited to fine-tuning; it also manifests during reinforcement learning (RL) and decoding. When we optimize a model to satisfy specific stylistic constraints—such as being polite, formal, or matching human preferences—we introduce competing objectives.
de Langis et al. highlight this fundamental tension in multi-style controllable generation. When the pressure to satisfy a style reward is high, the model's optimization landscape shifts: it begins prioritizing stylistic markers over factual grounding, leading to a degradation of internal knowledge retrieval.
This risk extends to preference alignment; Pedrotti et al. note that using Direct Preference Optimization (DPO) to shift a model's linguistic fingerprint toward human distributions can inadvertently compromise its reasoning and information retrieval capabilities.
Conversely, even when training specifically for factual consistency, Feng et al. find that models risk overfitting to the specific linguistic style of the training source (like news summaries), limiting their ability to generalize fact-checking to other domains.
Furthermore, manipulating decoding strategies to favor factuality introduces the reverse problem. As Jin et al. point out, manipulating logits to prioritize factual accuracy can negatively impact the stylistic versatility, instruction-following capabilities, or creative expression of the model. The two objectives behave like a zero-sum game.
Geometric Entanglement in Representation Space
Why are style and truth locked in this adversarial relationship? The answer lies within the geometry of the model's internal representation space.
In a standard transformer, the features corresponding to "style" (tone, persona, syntax) and "truth" (factual knowledge, semantic correctness) are not cleanly separated. Instead, they are geometrically entangled.
Entangled Space (Standard LLM) Disentangled Subspaces (StyliTruth)
^ Factual Truth ^ Factual Truth
/ |
/ <-- (Interference) | (Independent
/ | Steering)
+---------> Stylistic Persona +---------> Stylistic Persona
Shen et al. formalize this as "stylization-induced truthfulness collapse." Because the internal activation directions for style and truth interfere with one another, steering a model's hidden states toward a specific persona or tone inadvertently distorts the representations responsible for factual accuracy. When an activation vector is pushed along the "persona direction," it drifts away from the "truth direction," causing the model to lose its grip on factual constraints.
To break this tradeoff, researchers are exploring methods that avoid joint optimization. For example, Hakkani Tur et al. avoid naive fine-tuning by using an external preference agent to generate natural language rules, guiding a frozen model's style without corrupting its internal knowledge base.
Ultimately, resolving this conflict requires moving toward architectures and steering methods like those proposed by Shen et al., which explicitly disentangle style and truth into independent subspaces. Only by separating these internal representations can we build models that sound professional without sacrificing the truth.
Key Foundational Work
PAP2PAT: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairs (2025) — contributed 3 concept elements to this direction.
The Cocoruta Hub: Open and Curated Corpora, Datasets and Language Models on Brazilian Ocean Law (2025) — contributed 3 concept elements to this direction.
Derby LLM - Évaluation comparative des approches RAG et fine-tuning (2024) — contributed 3 concept elements to this direction.