TL;DR
This paper introduces Adversarial Noisy Instruction Tuning (ANIT), a novel method to improve Large Language Models' (LLMs) performance on Natural Language Understanding (NLU) tasks. ANIT leverages controlled noisy instructions and adversarial training to enhance semantic robustness, leading to more accurate NLU without requiring extensive high-quality data.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable success in Natural Language Generation (NLG), largely due to instruction tuning, which aligns their outputs with human preferences. However, their performance in Natural Language Understanding (NLU) tasks lags behind. A key challenge in improving NLU performance via instruction tuning is the scarcity of large-scale, high-quality instruction datasets. While data augmentation seems like a solution, even minor noise or degradation in data quality can significantly harm LLM performance. Creating specialized NLU instruction datasets is resource-intensive, making it a bottleneck. This paper aims to address this by developing a more efficient and robust method to enhance NLU capabilities using instruction tuning, balancing cost and performance.
Methods
The core of this paper is the Adversarial Noisy Instruction Tuning (ANIT) method, which comprises three main components:
- Noisy Instruction Construction: The authors generate noisy instructions by introducing controlled randomness. This includes techniques like random truncation, shuffling, insertion, and replacement to mimic real-world data imperfections. They also employ "in-context padding" by embedding extraneous data within instructions and "opposite noise" by confusing instructions with ambiguous terms. The goal is to create instructions that are slightly distorted but still interpretable, forcing the LLM to focus on the core task.
- Semantic Distortion Quantification: To manage the noise, ANIT quantifies the semantic distortion introduced. Two methods are used:
- Rule-Based Evaluation (RBE): Uses the Levenshtein distance to measure word-level differences between noisy and raw instructions.
- LM-Based Evaluation (LME): Leverages OpenAI's
text-embedding-3-largemodel to compute cosine similarity between the embeddings of noisy and raw instructions, providing a more precise estimate of semantic distortion. A scaling factor, $\beta$, is derived from these quantifications, indicating the intensity of semantic distortion. - Noise Response Method for Adversarial Training (NRM): This is a novel adversarial training strategy. It uses a two-step process:
- Step 1: Samples an initial perturbation $\delta_0$ from a uniform distribution, scaled by $(1-\beta)$. This adaptive scaling adjusts the adversarial training intensity based on the quantified semantic distortion: less intense training for more distorted instructions and vice-versa.
- Step 2: Enforces a constraint on the perturbation $\delta_1$ to ensure it stays within a predefined bound $\epsilon$. The adversarial loss is then added as a regularization term to the standard instruction tuning loss, encouraging the LLM to be robust against semantic distortions.
Datasets
The experiments were conducted on four representative NLU tasks:
- Named Entity Recognition (NER): CoNLL2003 and Ontonotes.
- Relationship Extraction (RE): SciERC and NYT.
- Text Classification (TC): SST2 and AGNews.
- Aspect-Based Sentiment Analysis (ABSA): 14Lap and 14Rest.
Instructions for these tasks were presented in both concise and detailed versions.
Key Insights and Ideas
- Semantic Robustness is Key for NLU: The paper posits that enhancing an LLM's "semantic robustness"—its ability to maintain consistent understanding across variations—is crucial for improving NLU performance. ANIT directly targets this by training the model to handle noisy instructions.
- Controlled Noise Can Be Beneficial: Contrary to the intuition that noise degrades performance, ANIT demonstrates that carefully controlled and quantified noise can actually improve NLU capabilities by forcing the model to focus on essential semantic information.
- Adaptive Adversarial Training: The Noise Response Method (NRM) is a significant contribution, offering an adaptive approach to adversarial training. By scaling perturbations based on semantic distortion, it dynamically adjusts the training intensity, making it more efficient and effective than fixed adversarial methods.
- LM-Based Evaluation is More Effective for NLU: The study suggests that LM-based evaluation methods are better suited for quantifying semantic distortion in the context of NLU tasks compared to rule-based methods, leading to more refined model performance.
Key Results
- Consistent Performance Gains: ANIT consistently outperformed baseline instruction tuning across various LLMs (Gemma-2B, LLaMA2-7B, LLaMA3-8B) and NLU tasks, regardless of whether detailed or concise instructions were used.
- 'Rand Replacement' is Often Most Effective: Among the noise variants, "Random Replacement" proved to be the most effective in improving performance.
- Generalization Capabilities: ANIT demonstrated strong generalization, maintaining performance gains across different datasets and tasks, even in few-shot scenarios. For instance, using only 5% of training data, ANIT showed an average performance improvement of 2.47%.
- Efficiency: While ANIT introduces some additional time cost compared to direct instruction tuning, it is significantly more time-efficient than other adversarial training methods like FreeLB, PGD, and FGSM, while achieving superior performance.
- Noise Strength Impact: The noise strength factor ($\omega$) plays a role, with optimal performance typically observed for values between 0.1 and 0.3. Performance declines when $\omega$ exceeds 0.3, but still remains above the baseline.
Limitations and Future Work
- Potential Limitations of Noise Variants: The paper acknowledges that certain noise variants might lead to performance degradation on specific tasks with detailed instructions. This could be attributed to the LLM's pre-training strategies or the specific construction of instructions.
- Hyperparameter Tuning: The effectiveness of ANIT relies on hyperparameters like $\omega$ and $\beta$, which require careful tuning.
- Future Work: The authors suggest further exploration into the impact of semantic distortion quantification on ANIT performance and investigating the effectiveness of LM-based evaluation for NLU tasks. They also highlight the potential for ANIT to be applied to a wider range of downstream tasks.
Summary updated