Sample Efficient Alignment Learning With Episodic Control

Van Dai Do, Quan Hung Tran, Ahmed Kirmani, Lu Zhang, Hung LeEMNLP 2025

TL;DR

This paper introduces ALEC, a novel framework for aligning Large Language Models (LLMs) with human preferences without requiring fine-tuning. ALEC uses an episodic memory to store and retrieve learned associations, enabling efficient alignment even with limited feedback, outperforming existing methods on harmlessness, helpfulness, and summarization tasks.

What Problem Does This Paper Address?

Aligning LLMs with human preferences and values is crucial but challenging, especially when access to feedback signals for guiding the model is limited. Traditional methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Policy Optimization (DPO) require extensive fine-tuning and large amounts of preference data, which are often costly and scarce. Inference-time methods exist but often rely on training parametric value models, which struggle when reward model (RM) calls are restricted or when dealing with limited preference data, leading to overfitting or slow execution. This paper addresses the need for efficient alignment learning methods that minimize reliance on expert feedback and are practical in low-resource scenarios.

Methods

The paper proposes Alignment Learning with Episodic Control (ALEC), a training-free framework. ALEC operates in two phases:

  • Memory Construction:
    • It leverages Episodic Control (Blundell et al., 2016), treating each prompt-response sequence as an episode.
  • A key-value memory stores associations between generated text (represented by embeddings) and their estimated values.
  • A novel confidence-based writing scheme is introduced to update stored values, optimizing the sample order for memory writing and updating neighboring states to reduce bias. This scheme aims to maximize the use of available data and improve value estimation accuracy.
  • An ε-greedy policy is used to select actions during generation to encourage exploration of diverse and high-quality actions.
  • A sample selection strategy based on an "insert score" (combining estimated value and uncertainty) is used to prioritize samples for memory storage.
  • Inference (Memory Read):
    • During decoding, ALEC uses the memory as a non-parametric nearest-neighbor estimator to approximate the expert reward model.
  • It estimates the values of candidate continuations and selects the one with the highest estimated value.
  • A confidence-based mechanism (is_certain) determines whether to rely on the memory's estimation or the base LLM's probabilities, especially when the memory is uncertain.
  • This process guides generation by selecting tokens that maximize estimated values, leading to improved alignment without modifying the LLM's parameters.

Datasets

The paper evaluates ALEC on the following datasets:

  • Anthropic HH: A standard benchmark for alignment learning, used to create two tasks:
  • helpful-base: Focusing on aligning responses to be maximally helpful.
  • harmless-base: Focusing on aligning responses to be maximally harmless.
  • TL;DR: A summarization task involving Reddit posts paired with summaries and human preference annotations.

Key Insights and Ideas

  • Non-parametric alignment is viable and effective: ALEC demonstrates that alignment can be achieved efficiently without the need for fine-tuning LLMs, relying instead on memory-based retrieval.
  • Episodic memory as a powerful tool: Storing and retrieving past experiences (prompt-response-value associations) allows the model to learn and adapt its generation strategy at inference time.
  • Confidence-based decision-making: The is_certain mechanism is crucial for balancing the reliance on the memory versus the base LLM, preventing over-reliance on potentially noisy memory estimates.
  • Data efficiency is key: The proposed confidence-based writing scheme and sample selection strategy are designed to maximize the utility of limited feedback, making ALEC suitable for low-resource scenarios.
  • Semantic similarity matters for memory retrieval: The use of nearest neighbors for memory lookup is shown to be more effective than farthest neighbors, highlighting the importance of semantically relevant past experiences.

Key Results

  • Superior performance: ALEC consistently outperforms state-of-the-art baselines (including fine-tuning and other inference-time methods) across harmless, helpful, and summarization tasks.
  • High win rates: On the harmless dataset with Vicuna-7B-v1.5, ALEC achieved an 11.74% gain over the second-best method. It also ranked first for Llama-2-7B-chat-hf and Mistral-7B-Instruct-v0.2.
  • Competitive with fine-tuned models: ALEC often surpasses other inference-time baselines and even rivals fine-tuned models like DPO in certain settings.
  • Improved average reward: ALEC significantly increases the average reward across benchmarks compared to the base model and other baselines.
  • Scalability: The method shows strong performance even with larger LLMs (13B parameters) and demonstrates minimal memory consumption, making it efficient for deployment.
  • Limited RM calls: ALEC achieves these results with a strict budget of 1000 RM calls, highlighting its sample efficiency.

Limitations and Future Work

  • Medium-scale datasets: The current evaluations are limited to medium-scale datasets. Further empirical evaluations are required to assess ALEC's scalability in more complex, large-scale datasets.
  • Hyperparameter tuning: While the paper provides default hyperparameters, further tuning, especially for the ε-greedy parameter, could potentially improve performance.
  • Future work: The authors suggest exploring more elaborate memory footprints and investigating the impact of different memory sizes and neighbor selection strategies. They also highlight the potential for ALEC to deliver high-quality alignment with minimal RM interactions, suggesting its broad applicability.

Summary updated

BibTeX
@inproceedings{do2025sample,
  author = {Van Dai Do and Quan Hung Tran and Ahmed Kirmani and Lu Zhang and Hung Le},
  title = {{Sample Efficient Alignment Learning With Episodic Control}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure