CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models

Son The Nguyen, Niranjan Uma Naresh, Theja TulabandhulaNAACL 2024

TL;DR

This paper introduces CURATRON, a novel framework for generating complete and robust preference data for aligning Large Language Models (LLMs). It tackles the critical issues of incomplete and adversarially corrupted human feedback, which can significantly degrade LLM alignment quality. CURATRON offers a robust solution that can recover accurate rankings even with noisy and partial data, paving the way for more reliable LLM alignment.

What Problem Does This Paper Address?

Aligning Large Language Models (LLMs) with human values and expectations is crucial for their safe and effective deployment. The dominant method for this alignment is preference learning (PL) using human feedback. However, gathering this feedback is a slow, expensive, and often imperfect process. Human participants might intentionally provide inaccurate or harmful feedback, and the resulting datasets are frequently incomplete, meaning not all possible pairs of responses are compared. These issues pose a significant challenge to the integrity and reliability of LLM alignment, especially as the scale of models and data grows. This paper addresses the need for methods that can handle both incomplete comparison data and adversarial noise in the feedback.

Methods

The paper proposes two main algorithmic contributions:

  • RORATRON (Robust Preference Data for Rigorous Alignment): This algorithm is designed to handle adversarial noise in pairwise comparison data. It leverages Robust PCA (RPCA) to denoise the preference matrix and then uses a pairwise ranking procedure to recover the true ranking of responses. RORATRON is proven to provide high-probability learnability with a small margin of error, even in the presence of adversarial noise.
  • CURATRON (Complete Robust Preference Data for Rigorous Alignment): This algorithm extends RORATRON to address scenarios where the preference data is both incomplete (missing comparisons) and adversarially corrupted. It combines matrix completion techniques (specifically, the 'OptSpace' algorithm) with robust PCA to first fill in missing entries and then denoise the resulting matrix, ultimately leading to a robust ranking.

The core idea behind both methods is to transform the preference data into a low-rank matrix (often after applying a logit function) and then use robust matrix recovery techniques to handle noise and missing entries.

Datasets

The paper does not introduce new datasets. Instead, it uses existing LLM preference data for evaluation. Specifically, in their experiments, they utilize data from the MT-Bench dataset (Zheng et al., 2023), which includes responses from various LLMs like GPT-3.5, GPT-4, Claude, Vicuna, Alpaca, and Llama, among others. They then use OpenAI's GPT-4 Turbo to generate a ground truth ranking for this data.

Key Insights and Ideas

  • Adversarial noise significantly degrades ranking quality: The paper demonstrates that even a small amount of adversarial noise can lead to a substantial degradation in the accuracy of estimated rankings, highlighting the necessity of robust methods.
  • Low-rank structure is key: The underlying assumption is that the true preference data, when transformed (e.g., via a logit function), often exhibits a low-rank structure. This property is exploited by Robust PCA to separate the true signal from noise.
  • Matrix completion and robust PCA are a powerful combination: By integrating matrix completion for missing data with robust PCA for noise reduction, CURATRON effectively handles both challenges simultaneously, leading to more reliable alignment.
  • Human feedback doesn't need to be perfect: The experiments suggest that even with a significant portion of missing or corrupted comparisons, CURATRON can still recover a good ranking, implying that collecting exhaustive and perfectly clean human feedback might not always be necessary.

Key Results

  • Theoretical Guarantees: The paper provides theoretical guarantees for both RORATRON and CURATRON, showing that they can recover the true ranking with high probability and a small error bound, even under adversarial conditions and with missing data. For instance, Theorem 1 states that with a sufficient number of comparisons, Algorithm 1 (RORATRON) returns an estimated permutation $\hat{\sigma}$ such that $dist(\hat{\sigma}, P) \leq \epsilon$. Theorem 2 provides similar guarantees for CURATRON in the partially observed setting.
  • Experimental Validation: In experiments with $n=15$ responses, CURATRON successfully reconstructed low-rank preference matrices even with 10% adversarial corruption and 10% unobserved comparisons. The reconstructed rankings closely matched the original rankings.
  • Performance under varying noise levels: The experiments showed that CURATRON could achieve low Normalized Frobenius Error (NFE) and high correlation/ranking accuracy even when up to 30% of the total comparisons were either missing or adversarially corrupted. For example, with 15% missing and 15% adversarial data, NFE was around 4%.

Limitations and Future Work

  • Computational Complexity: While the algorithms are theoretically sound, the paper notes that the overall computational complexity for RORATRON can be $O(n^4 \log^2 n/\epsilon)$. Future work could focus on improving this efficiency.
  • Assumptions on Adversary: The current work assumes a specific model for adversarial noise (additive, skew-symmetric sparse matrix). Exploring other notions of adversarial noise is a potential future direction.
  • Extension to other models: The paper primarily focuses on BTL and LR models. Extending the approach to other preference learning models or different types of LLM alignment tasks is a natural next step.
  • Investigating Model Capacity: The authors mention an interest in examining the relationship between robust PL and model capacity, and how this impacts generalization performance.
  • Entity Corruption: The paper suggests investigating whether the approach can be extended to solve entity corruption problems in retrieval models.

Summary updated

BibTeX
@inproceedings{nguyen2024curatron,
  author = {Son The Nguyen and Niranjan Uma Naresh and Theja Tulabandhula},
  title = {{CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure