Collaborative Alignment of NLP Models

Fereshte Khani, Marco Tulio RibeiroNeurIPS 2023 · OpenReview

TL;DR

CoAlign is a new framework that allows multiple users to collaboratively "teach" specific concepts and rules to NLP models. By training small "local" models for each concept and using an LLM to hunt for disagreements between these experts and the main global model, CoAlign effectively fixes model bugs and enforces business rules while preventing new updates from interfering with existing knowledge.

What Problem Does This Paper Address?

Even the most advanced NLP models often fail to follow specific business rules or align with human values—for example, a sentiment analysis model might incorrectly associate "I am Muslim" with a negative sentiment.

The standard fix is to add more training data. However, this approach has three major pitfalls:

  1. Shortcuts: Models might learn a "cheat" (e.g., "any sentence with a religious word is neutral") rather than the actual concept.
  2. Overfitting: The model might only learn the specific examples provided and fail to generalize.
  3. Interference: Fixing one problem often breaks something else. Adding data to fix religious bias might accidentally degrade the model's ability to detect actual "love" or "hate" in other contexts.

Furthermore, a single developer can rarely think of every possible edge case. There is a need for a system that allows multiple users to define concepts collaboratively without stepping on each other's toes.

Methods

The researchers introduced CoAlign, which treats model alignment as a collaborative, iterative loop. The core innovation lies in the interplay between Local and Global models.

CoAlign Framework
  • Local Concept Models: For every new concept (e.g., "Downward Monotonicity" or "Religious Neutrality"), the system trains a simple "local expert" model on a small set of user-provided examples.
  • The Global Model: This is the main model that integrates the original training data with all the new concepts provided by various users.
  • The Disagreement Loop: Instead of asking users to come up with thousands of examples, CoAlign uses a Large Language Model (like GPT-3) to generate new instances. Crucially, it prioritizes generating examples where the local expert and the global model disagree.
  • Interference Management: When a new user adds a concept, the system checks the updated global model against the local experts of all previous concepts. If a disagreement is found, it acts like a "regression test" in software engineering, prompting the user to resolve the conflict.
Geometric Intuition of CoAlign

Datasets

The authors evaluated CoAlign across several tasks and datasets:

  • Quora Question Pairs (QQP): Used to identify and fix linguistic bugs (e.g., understanding that "Is X an artist?" is different from "Is X a famous artist?").
  • MNLI (Multi-Genre Natural Language Inference): Used to teach complex logical concepts like upward and downward monotonicity.
  • Amazon Product Reviews: Used for sentiment analysis across different product categories.
  • Biased-SB: A synthetic dataset created by the authors to test if the system can recover from extremely biased initial user input.

Key Insights and Ideas

  • Local Simplicity: The authors rely on the principle that while a global model is complex, model behavior in a "local region" (around a specific concept) is often much simpler and can be captured by a "cheap" local expert.
  • Disagreement as a Probe: The most valuable data points aren't random; they are the ones where the "expert" and the "generalist" clash. This is where the model's boundaries are fuzzy.
  • Alignment as Software Engineering: By maintaining local models for every concept, the authors treat model alignment like a codebase. You can't "merge" a new concept if it breaks the "tests" (local models) of previous concepts.
  • Overcoming User Bias: Surprisingly, the system works even if the user provides a very biased starting set. Because the LLM-driven generator looks for disagreements with the base model, it naturally pushes the user to label examples that challenge their initial bias.

Key Results

  • Better than SOTA: CoAlign outperformed AdaTest (a leading model debugging tool) by finding more diverse bugs and fixing them with significantly less interference to the rest of the model.
  • Interference Reduction: In head-to-head tests, CoAlign was much better at ensuring that learning a new concept didn't "break" the original behavior of the model compared to standard data collection.
  • Efficiency: The disagreement-based sampling was shown to be more effective at reaching high accuracy than standard active learning techniques like "uncertainty sampling" or random selection.
  • Human-in-the-loop: A pilot study showed that the process actually helped users clarify their own definitions of concepts as they interacted with the model's "confusions."

Limitations and Future Work

  • Task Variety: The study focused primarily on classification tasks. Extending this to generative tasks (like open-ended chatbots) would require using local classifiers to judge if a generated response aligns with a concept.
  • Literal Disagreements: The framework assumes there is one "true" label. It doesn't yet have a formal mechanism to resolve cases where two users fundamentally disagree on what the correct label should be.
  • Malicious Actors: There are currently no safeguards against a user intentionally trying to "align" the model toward harmful or biased behavior.
  • Theoretical Scope: While the paper provides a theoretical bound for linear models, a more general theoretical analysis for deep neural networks remains an open challenge.

Summary updated

Abstract

Despite substantial advancements, Natural Language Processing (NLP) models often require post-training adjustments to enforce business rules, rectify undesired behavior, and align with user values. These adjustments involve operationalizing "concepts"—dictating desired model responses to certain inputs. However, it's difficult for a single entity to enumerate and define all possible concepts, indicating a need for a multi-user, collaborative model alignment framework. Moreover, the exhaustive delineation of a concept is challenging, and an improper approach can create shortcuts or interfere with original data or other concepts. To address these challenges, we introduce CoAlign, a framework that enables multi-user interaction with the model, thereby mitigating individual limitations. CoAlign aids users in operationalizing their concepts using Large Language Models, and relying on the principle that NLP models exhibit simpler behaviors in local regions. Our main insight is learning a \emph{local} model for each concept, and a \emph{global} model to integrate the original data with all concepts. We then steer a large language model to generate instances within concept boundaries where local and global disagree. Our experiments show CoAlign is effective at helping multiple users operationalize concepts and avoid interference for a variety of scenarios, tasks, and models.
BibTeX
@inproceedings{khani2023collaborative,
  author = {Fereshte Khani and Marco Tulio Ribeiro},
  title = {{Collaborative Alignment of NLP Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=f39Q3JyoIi},
}
Figure