Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings

César González-Gutiérrez, Dirk HovyarXiv 2025 · DOI · arXiv

Abstract

Does a "good" prompt actually improve a language model's internal "thoughts" about a task? This study probes the embeddings of models like BERT, RoBERTa, and GPT-2 to determine if task-relevant prompts enhance internal representations. The findings reveal a "relevance paradox": while prompts reshape embedding spaces, relevant prompts do not consistently outperform random or irrelevant ones in task alignment. Furthermore, the effect is highly architecture-dependent, with GPT-2 often showing degraded representation quality when prompted.

Introduction

In-Context Learning (ICL) and prompting have become the standard paradigm for utilizing Large Language Models (LLMs) in zero-shot scenarios. While it is well-documented that prompt selection drastically alters model output, the underlying mechanism at the representation level remains poorly understood. A common assumption is that relevant prompts (e.g., "Classify this sentiment:") help the model contextualize input, creating more distinct internal representations that facilitate task resolution. This paper empirically tests this assumption: Does the quality of a model's internal embeddings actually improve with task-relevant prompts compared to irrelevant ones or no prompt at all?

Methodology

The authors conducted probing experiments on BERT, RoBERTa, and GPT-2 to analyze internal representations across four major NLP tasks: Toxicity Detection (Wiki Toxic), Sentiment Analysis (IMDB), Topic Classification (e.g., AG News, Arise News), and Natural Language Inference (RTE, Adv. NLI).

  1. Prompting Framework: Input samples were wrapped in 26 different templates, including five task-relevant prompts, five random word sequences (to control for length and noise), and a "no prompt" baseline.
  2. Embedding Extraction: Sentence-level embeddings were generated using various pooling strategies, such as averaging token activations or utilizing the [CLS] token.
  3. Probing: A simple MaxEnt classifier (the "probe") was trained on these embeddings. Higher probe performance indicates a more task-informative internal representation.
  4. Task Alignment: The study also measured "Task Alignment," a metric quantifying how well the geometry of the embedding space matches actual task labels.

Key Findings

The Relevance Paradox

Surprisingly, "relevant" prompts are not guaranteed to produce superior representations. In many instances, prompts for unrelated tasks—or even random word sequences—improved probe performance as much as, or more than, the "correct" prompt. This suggests that the mere presence of additional context can reshape the embedding space, but not necessarily in a way that aligns with the intended task logic.

Probing performance on the RTE benchmark (NLI) using BERT, showing the variance in representation quality across different prompt types.

Model-Specific Sensitivity

The impact of prompting is highly inconsistent across architectures. While BERT showed modest gains with relevant prompts, GPT-2 frequently exhibited degraded representation quality when any prompt was added compared to the raw input baseline.

Probing F1 scores for Wiki Toxic across BERT, RoBERTa, and GPT-2 variants, illustrating the inconsistent impact of prompting.

Contextualization and Reshaping

The study found that "Static Prompts" (adding a prompt vector to a sample vector) had no effect, proving that reshaping occurs through the model's internal attention mechanism. Prompts definitely move representations within the embedding space (reshaping), but they do not always move them into a more separable configuration (improving).

Quantitative Results

  • BERT + Toxicity: A relevant prompt improved the probing F1 score from 60.65% (no prompt) to 61.57%. However, a random prompt also reached 61.11%.
  • GPT-2 Performance: On the Wiki Toxic dataset, GPT-2's "no prompt" baseline was its best-performing configuration (60.58%). Adding any prompt generally lowered the quality of the representation for the probe.
  • Correlation: A strong positive correlation (Pearson’s $r = 0.7475$) was found between probing performance and Task Alignment, confirming that when prompts do help, they do so by physically reorganizing the embedding space to match the task's logic.

Conclusion and Future Work

This study highlights that prompting does not universally "improve" representations in a task-aligned manner. The authors suggest that the base models used (BERT, GPT-2) lack the instruction fine-tuning necessary to properly align representations with human-written instructions. Future work should explore these dynamics in larger, instruction-tuned models and investigate layer-wise dynamics to understand how representations evolve through transformer blocks.

Summary updated

BibTeX
@misc{gonzalezgutierrez2025do,
  author = {César González-Gutiérrez and Dirk Hovy},
  title = {{Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings}},
  year = {2025},
  eprint = {2510.19694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.19694},
}
Figure