BeFA: A General Behavior-driven Feature Adapter for Multimedia Recommendation

Qile Fan, Penghang Yu, Zhiyi Tan, Bing-Kun Bao, Guanming Lu

Introduction

Modern multimedia recommendation systems, powering platforms like TikTok and Pinterest, increasingly rely on pre-trained multi-modal encoders such as CLIP or ImageBind. While these encoders are powerful generalists, they often fail to capture the specific nuances of user preference. Researchers have introduced the Behavior-driven Feature Adapter (BeFA), a plug-and-play module designed to bridge the gap between general content features and specific user behaviors. By using interaction history as a guiding signal, BeFA filters out irrelevant "noise" in multimedia content, leading to significant performance improvements in state-of-the-art recommendation models.

The Core Challenge: Information Drift and Omission

The primary limitation of using off-the-shelf encoders is that they extract features from an entire image or video simultaneously, without context for the user's intent. This leads to two critical issues:

  1. Information Drift: The encoder focuses on irrelevant background elements. For instance, if a user is interested in a specific dress, a general encoder might prioritize the model's face or the park setting in the background.
  2. Information Omission: The encoder may completely overlook subtle details that drive user interest, such as the specific texture of a fabric or the intricate pattern of a ring.
Figure 1: Conceptual illustration of Information Drift where the encoder prioritizes background features over user-relevant product attributes.

Because these noisy features are subsequently fused with behavioral data, the resulting user preference models are often suboptimal, as they are built on a foundation of irrelevant content data.

Proposed Method: The BeFA Framework

The researchers proposed a two-pronged approach to rectify these issues: a visualization tool to diagnose the problem and a parameter-efficient adapter to solve it.

1. Similarity-based Attribution Analysis

To confirm the existence of drift and omission, the authors developed a novel visualization tool. Unlike standard Class Activation Mapping (CAM) which identifies features used for classification, this tool generates heatmaps showing which parts of an image are most similar to the user's behavioral features (interaction history).

Figure 2: Heatmaps generated by attribution analysis demonstrating Information Drift and Information Omission across different multimedia items.

As seen in the heatmaps, standard encoders frequently "drift" toward backgrounds or irrelevant subjects, visually validating the need for a behavior-driven refinement process.

2. The Behavior-driven Feature Adapter (BeFA)

BeFA is designed to be inserted into existing pipelines without retraining the massive underlying encoders. It operates through three distinct stages:

  • Decoupling: Original content features are projected into a "decoupled space" to separate various semantic components.
  • Filtering: A gating mechanism, guided by behavioral information, acts as a "lens" to select only the decoupled features relevant to the user's past actions.
  • Reconstruction: The filtered features are recombined into a refined representation that aligns closely with actual user preferences.

Experimental Results

BeFA was evaluated against several top-tier recommendation models (including BM3, LATTICE, and MGCN) across three diverse datasets: TMALL (E-commerce), Microlens (short videos), and H&M (fashion).

Key findings include:

  • Significant Gains: BeFA improved Recall@20 by an average of 9.07% and NDCG@20 by 11.02% over baseline models.
  • Efficiency: The module is highly parameter-efficient, typically adding less than 2% to the total system parameters.
  • Robustness: The most dramatic improvements were seen in the Microlens dataset, demonstrating BeFA's ability to filter complex, noisy video backgrounds effectively.
  • Superiority to LoRA: BeFA consistently outperformed other popular tuning methods like Low-Rank Adaptation (LoRA) and Soft Prompt Tuning in handling information drift.

Conclusion and Future Work

BeFA demonstrates that user behavior is the ultimate ground truth for filtering multimedia content. While highly effective, the current model relies on a manually tuned hyperparameter for the decoupling space dimension. Future research aims to make this dimension a learnable parameter, allowing the system to adaptively adjust its complexity based on the specific dataset and application scenario.

Summary updated

BibTeX
@inproceedings{fan2025befa,
  author = {Qile Fan and Penghang Yu and Zhiyi Tan and Bing-Kun Bao and Guanming Lu},
  title = {{BeFA: A General Behavior-driven Feature Adapter for Multimedia Recommendation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i11.33266},
}
Figure