Knowledge-Decoupled Synergetic Learning: An MLLM based Collaborative Approach to Few-shot Multimodal Dialogue Intention Recognition

Bin Chen, Yu Zhang, Hongfei Ye, Yizi Huang, Hongyang ChenWWW 2025

Introduction

In the domain of e-commerce, intelligent customer service systems must interpret multimodal inputs, such as user-provided screenshots alongside textual queries. While Multimodal Large Language Models (MLLMs) have advanced significantly, they frequently encounter a "seesaw effect" during joint fine-tuning. This phenomenon occurs when optimizing a model for one task (e.g., image scene recognition) inadvertently degrades its performance on another (e.g., dialogue intent recognition) due to weight interference in the shared parameter space. This interference creates a performance ceiling for monolithic models, as parameter updates for disparate tasks conflict within the model's internal matrices.

Methodology: The KDSL Framework

To address these challenges, we introduce Knowledge-Decoupled Synergetic Learning (KDSL). KDSL moves away from a single-model approach toward a collaborative architecture that decouples explicit domain logic from latent neural representations.

1. Knowledge Decoupling via Rule Engine

The framework utilizes a smaller MLLM (Qwen2-VL-2B) as a dedicated "rule generator." By employing Monte Carlo Tree Search (MCTS), the system explores and validates logical rules (e.g., "If the image contains a tracking number and the text mentions 'location', the intent is 'Logistics Query'"). These rules are evaluated for quality using a Reward Score and stored in an interpretable Rule Base.

2. Supervised Fine-Tuning and Collaborative Reasoning

A larger model (Qwen2-VL-7B) is fine-tuned on domain-specific datasets from Taobao, utilizing data augmentation techniques such as horizontal flipping. During inference, the system performs collaborative reasoning: the Rule Engine acts as a specialist for explicit logic, correcting or refining the 7B model's predictions when specific visual or textual markers are identified.

The KDSL framework architecture, highlighting the MM-RuleEngine and the collaborative reasoning process.

Experimental Results

The KDSL framework was evaluated on two real-world datasets from Taobao, comprising 10,000 samples each. Performance was measured using class-weighted F1 scores across three metrics: Dialogue Intention Score (DIS), Image Scene Score (ISS), and a unified Online Submit Score (OSS).

Performance Comparison

KDSL significantly outperformed standard fine-tuning methods and larger standalone models. On Test Set 1, KDSL achieved an OSS of 87.48%, a 6.37% improvement over the baseline Qwen2VL-7B-FT. Notably, the KDSL approach (combining 7B and 2B models) surpassed the larger InternVL-8B-FT model, proving that architectural synergy can be more effective than simply increasing parameter count.

Performance comparison of KDSL against various MLLM baselines across two test sets.

Key Insights

  • Weight Interference Mitigation: By decoupling explicit rules from the neural network's black-box parameters, KDSL effectively bypasses the "seesaw effect."
  • Small Model Utility: Smaller models (2B parameters) are highly effective at generating and self-assessing logical rules when guided by search algorithms like MCTS.
  • Interpretability: The use of a Rule Base provides a layer of transparency often missing in pure neural approaches, allowing for verifiable logic in customer service interactions.

Conclusion and Future Work

KDSL sets a new benchmark for multimodal dialogue systems by leveraging the strengths of both large-scale neural models and explicit logical reasoning. While the system currently depends on OCR accuracy (using LayoutLMv3) and involves upfront search costs for rule generation, it demonstrates superior performance in complex e-commerce tasks. Future research will explore more sophisticated OCR integration and extending the framework to multi-turn dialogue scenarios.

Summary updated

BibTeX
@inproceedings{chen2025knowledge,
  author = {Bin Chen and Yu Zhang and Hongfei Ye and Yizi Huang and Hongyang Chen},
  title = {{Knowledge-Decoupled Synergetic Learning: An MLLM based Collaborative Approach to Few-shot Multimodal Dialogue Intention Recognition}},
  booktitle = {The Web Conference},
  year = {2025},
}
Figure