It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension

Sagi Shaier, Lawrence Hunter, Katharina von der Wense

In the rapidly evolving landscape of Large Language Models (LLMs), researchers often focus on scaling parameters or refining training data. However, recent research reveals that performance on Reading Comprehension (RC) tasks is remarkably sensitive to a much simpler factor: prompt structure. By optimizing the order of inputs and applying basic emphasis markers, models can achieve accuracy gains of up to 31%, frequently allowing smaller models to outperform their much larger counterparts.

The Problem: Lack of Standardized Prompting

In a typical Reading Comprehension task, a model is provided with a context (such as a Wikipedia passage) and a question about that context. Despite the ubiquity of this task, there is no industry standard for how to present these inputs. Some researchers place the question first, while others lead with the context. This paper investigates whether this ordering matters and if "nudging" the model to pay attention to specific prompt segments can improve performance.

Methodology: Ordering and Emphasis

The study evaluated nine different LLMs, ranging from GPT-2 to Llama-2 13B, across three diverse benchmarks: Natural Questions (NQ), SQuAD v1.1, and AdversarialQA. The researchers tested two primary strategies:

  1. Input Ordering: Comparing "Question First" (Question: [Q] Context: [C]) against "Context First" (Context: [C] Question: [Q]).
  2. Emphasis Strategies:
    • Marked Prompting (MP): A simple approach wrapping the question or context in special characters or tags, such as *question*, "question", or <emphasize>question</emphasize>.
  3. Attention Steering (AS): A sophisticated method that manually down-weights the attention scores of tokens outside the target string during the model's inference process.
Visualizing the different prompt configurations: Question First vs. Context First, and various emphasis targets.

Key Results and Performance Gains

The impact of these simple structural changes was profound. Across almost all models and datasets, the "Context-First" rule held true: placing the context before the question significantly improved accuracy. On average, this swap resulted in a 13.46% accuracy increase. In the most extreme case—GPT-J on the Natural Questions dataset—accuracy jumped from 33.3% to 64.5% simply by changing the input order.

Emphasis also proved to be a powerful lever. Using the best emphasis method (typically the <emphasize> tag) provided accuracy improvements of up to 36% in specific settings.

Detailed accuracy results showing the impact of ordering and emphasis across multiple LLMs and benchmarks.

Why It Works: Perplexity and Parametric Knowledge

The researchers used Perplexity to measure how "confusing" different prompt structures were to the models. They found that Context-First prompts showed an average perplexity reduction of 0.73, suggesting that LLMs find it easier to process a question once they have already ingested the relevant background information.

Furthermore, the study highlighted a critical distinction between Known and Unknown knowledge. Emphasis is most effective when the model does not already know the answer from its internal training data (parametric knowledge). When a fact is new to the model, these structural cues act as a vital guide for the model's attention.

Insights and Future Directions

One of the most surprising findings was that simple Marked Prompting was often just as effective as the more complex Attention Steering. This suggests that models are highly sensitive to surface-level formatting cues.

While this study focused on Reading Comprehension, it raises questions about whether similar rules apply to tasks like summarization or reasoning. The authors conclude by calling for a standardization of RC input presentation to ensure fair and consistent benchmarking across the NLP community. For practitioners, the takeaway is clear: before reaching for a larger model, try putting your context first and wrapping your key instructions in <emphasize> tags.

Summary updated

Abstract

Natural language processing has seen rapid progress over the past decade. Due to the speed of developments, some practices get established without proper evaluation. Considering one such case and focusing on reading comprehension, we ask our first research question: 1) How does the order of inputs – i.e., question and context – affect model performance? Additionally, given recent advancements in input emphasis, we ask a second research question: 2) Does emphasizing either the question, the context, or both enhance performance? Experimenting with 9 large language models across 3 datasets, we find that presenting the context before the question improves model performance, with an accuracy increase of up to 31%. Furthermore, emphasizing the context yields superior results compared to question emphasis, and in general, emphasizing parts of the input is particularly effective for addressing questions that models lack the parametric knowledge to answer. Experimenting with both prompt-based and attention-based emphasis methods, we additionally find that the best method is surprisingly simple: it only requires concatenating a few tokens to the input and results in an ac- curacy improvement of up to 36%, allowing smaller models to outperform their significantly larger counterparts.
BibTeX
@inproceedings{shaier2024it,
  author = {Sagi Shaier and Lawrence Hunter and Katharina von der Wense},
  title = {{It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.491},
}
Figure