Speak Like a Native: Prompting Large Language Models in a Native Style

Zhicheng Yang, Yiwei Wang, Yinya Huang, Jing Xiong, Xiaodan Liang, Jing Tang

Introduction: The Style Gap in LLM Prompting

In-context learning (ICL) has become the standard method for guiding Large Language Models (LLMs) through complex tasks. By providing a few examples of step-by-step reasoning—known as Chain-of-Thought (CoT) prompting—researchers have significantly improved model performance on mathematical and logical problems. However, these examples are typically written by humans, creating a "style gap."

The concept of native style: LLMs perform better when prompted in their own linguistic style.

LLMs are highly sensitive to the specific wording and structure of prompts. Forcing a model to follow human-style reasoning can be counterproductive, much like asking a native speaker to follow a clunky translation. This research introduces AlignedCoT, a method that replaces human-written demonstrations with "native-speaking" examples—reasoning chains generated by the model itself and then refined for accuracy.

The AlignedCoT Methodology

To bridge the style gap, the authors propose a three-step pipeline to construct "LLM-acquainted" prompts. This process ensures that the demonstrations provided to the model during few-shot learning are both factually correct and linguistically natural for the specific LLM being used.

Comparison between Vanilla CoT and the three-step AlignedCoT construction process.
  1. Probing: The LLM is first asked to solve the example questions in a zero-shot manner (e.g., using "Let's think step by step"). This captures the model's inherent reasoning style and linguistic patterns.
  2. Refining: Since zero-shot reasoning can be error-prone, an iterative refinement process is applied. If the model makes a mistake, a human or a stronger model corrects only the specific error, allowing the LLM to complete the rest of the thought. This "minimalist intervention" preserves the model's natural style while ensuring the final answer is correct.
  3. Formatting: The resulting reasoning steps are standardized into a consistent structure (e.g., using "STEP N:" and "Answer: [X]") to guide the model's output during inference.

Experimental Evaluation

The AlignedCoT method was evaluated across several rigorous benchmarks, including mathematical reasoning (GSM8K, AQUA, SVAMP) and commonsense reasoning (BIG-bench Penguins). The researchers tested the method on GPT-3.5-Turbo, GPT-4, and Llama-2-7b-chat.

Key Results

  • Performance Gains: GPT-3.5-Turbo saw an average accuracy boost of +3.2% across all benchmarks. GPT-4 improved by +1.7%, and Llama-2-7b also showed consistent benefits, proving the method's generalizability across model scales.
  • Logical Pitfall Detection: One of the most significant findings involved "logical paradoxes"—questions that are mathematically solvable but logically impossible. GPT-4's ability to recognize these errors jumped from 28.2% with standard CoT to 78.9% with AlignedCoT.
  • Synergy with RAG: By rewriting the GSM8K training set into the LLM's native style (creating the GSM8K-Align dataset), the authors found that retrieved examples became more effective teachers, improving performance by up to +3.6%.

Discussion and Insights

The core advantage of AlignedCoT lies in its ability to elicit stored knowledge more effectively by operating within the model's own internal representations. By not forcing the model to bridge the gap between human logic and its own linguistic habits, the prompt acts in harmony with the model's inherent stylistic nuances.

While the refinement process currently involves some manual oversight, the results suggest that "native style" is a fundamental property of LLMs. Future work may focus on fully automating the refinement step to reduce the computational and human cost of building these optimized prompts.

Conclusion

AlignedCoT demonstrates that how we talk to LLMs matters as much as what we tell them. By moving away from human-centric prompting toward a model-native approach, we can unlock higher levels of reasoning and accuracy, particularly in complex domains like mathematics and logic.

Summary updated

BibTeX
@inproceedings{yang2024speak,
  author = {Zhicheng Yang and Yiwei Wang and Yinya Huang and Jing Xiong and Xiaodan Liang and Jing Tang},
  title = {{Speak Like a Native: Prompting Large Language Models in a Native Style}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.163},
}
Figure