Abstract
Large Language Models (LLMs) are highly sensitive to input quality, where "noisy" or irrelevant context can lead to hallucinations and decreased performance. Traditional prompt engineering, while helpful, is often inconsistent and expert-dependent. This paper introduces Reinforced Token-Level Input Refinement (RTLIR), a plug-and-play module that automates the pruning of irrelevant tokens using reinforcement learning. By treating input refinement as a Markov Decision Process (MDP), RTLIR achieves an average accuracy improvement of 6% across various tasks and models.
Introduction
Despite their advanced reasoning capabilities, LLMs often struggle with redundant descriptions, distracting phrasing, or irrelevant context. Current mitigation strategies primarily rely on manual prompt engineering—instructing the model to "ignore irrelevant text." However, this approach is limited by human bias, expert dependency, and a lack of consistency across different model architectures.
The core challenge is automating the "cleaning" of input data at a fine-grained level. RTLIR addresses this by acting as a model-agnostic agent that filters the input sequence before it reaches the LLM, ensuring that only the most semantically relevant tokens are processed.
Methodology: Reinforced Token-Level Input Refinement
RTLIR formalizes input refinement as a Markov Decision Process (MDP) solved via Reinforcement Learning. The system operates at the token level, making binary decisions—Keep or Remove—for every token in the input sequence.
The RL Framework
The agent's state is defined by the concatenation of token embeddings and the history of previous actions. To guide the learning process, the authors implement a Dual-Reward System:
- Immediate Reward: This measures the cosine similarity between the current token and the target embedding (the desired answer), ensuring semantic relevance.
- Terminal Reward: This evaluates the change in the LLM’s output quality after the entire sequence is refined. It rewards the agent if the pruned text increases the probability of the correct answer compared to the original text.
The system utilizes a Double Q-Learning network with Prioritized Experience Replay. Because RTLIR operates on token embeddings, it is "plug-and-play," allowing it to be used with various LLMs (e.g., Llama, Qwen, Gemma) without requiring any retraining of the underlying language model.
Experimental Evaluation
The researchers evaluated RTLIR using two primary benchmarks: PAWS-X, for cross-lingual stability across six languages, and the Language Model Evaluation Harness, covering diverse domains from Ancient Chinese to Machine Learning.
Key Results
Experimental results demonstrate that "less is more" when it comes to LLM inputs:
- Accuracy Boost: Across all tested scenarios, RTLIR provided an average accuracy increase of 6%.
- Model-Specific Gains: The Llama-2-7B model saw a significant jump, with average accuracy rising from 40.03% to 51.06%.
- Cross-Lingual Success: The method proved effective across most languages in the PAWS-X dataset. For instance, English performance for Llama-2-7B improved from 31.25% to 45.35%.
- Qualitative Refinement: Case studies revealed that the agent successfully removed emojis, redundant adjectives, and repetitive phrases, resulting in cleaner, more formal inputs that facilitated better reasoning.
Discussion and Conclusion
RTLIR represents a shift from prompt-level instructions to token-level physical refinement. By removing the "fluff" before the LLM sees it, the system preserves the original text's core structure while maximizing clarity.
However, some limitations remain. The system showed a slight performance dip in French, likely due to unique linguistic structures not fully captured in the training set. Additionally, running an RL agent for every token introduces computational overhead during inference. Future work will focus on more complex reward functions and scaling the system to handle long-form documents where the "needle in a haystack" problem is most prevalent.
Summary updated