Do the Details Help the Whole? One-Way Transfer from Person Actions to Group Intent in VLMs on JRDB-Social

Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, QingHao Zhang, Shiwen NiarXiv 2026 · arXiv

TL;DR

In the world of social robotics, we often assume that understanding individual actions (the "details") and group intent (the "whole") are mutually reinforcing tasks. This paper reveals a surprising "one-way street": training Vision-Language Models (VLMs) to describe individual actions significantly improves their ability to predict group intent, but training on group intent does absolutely nothing to help them recognize individual actions.

What Problem Does This Paper Address?

When a mobile robot navigates a crowded campus, it needs to perceive the scene at multiple granularities. It must identify what individuals are doing (e.g., "looking at the robot," "holding something") and understand the broader social context of the group (e.g., "waiting for a bus," "socializing").

Historically, researchers building hierarchical models assumed these two levels of understanding supported each other. However, with the rise of general-purpose VLMs, the nature of this relationship has become a black box. The authors set out to rigorously test whether "the details help the whole" and vice versa, specifically looking for a "Mutual Reinforcement Effect" (MRE) where joint training improves performance on both tasks.

Methods

The researchers fine-tuned several versions of the Qwen3.5 (4B and 9B) and Qwen3-VL models using a variety of "training arms" to isolate the effects of multi-task learning.

Figure 2: Overview of the study. (a) The training arms differ only in which levels appear as targets and in which order. (b) Controls on the joint target, the prompt, the learning rate, the intent labels and the training steps. (c) Other backbones, a tool-using agent and GRPO rewards. (d) Analyses inside the trained models.

Key training strategies included:

  • Single-Task References: Training only on actions (S-F) or only on intent (S-C).
  • Joint Training: Writing both in one response, either actions then intent (J-FC) or intent then actions (J-CF).
  • Conditioned Training: Providing one level as a "hint" in the prompt to see if it helps the model predict the other.
  • Mechanistic Analyses: Using techniques like "component grafting" (swapping weights between models) and "forced prefixes" (forcing the model to start with a specific answer) to see where the knowledge actually resides.

To keep the science honest, they pre-registered a "verdict rule": a training method only "REINFORCES" if it beats both single-task references.

Datasets

The study utilized JRDB-Social, a dataset recorded from a mobile robot on a university campus. It features:

  • 27 action labels for individuals (e.g., walking, cycling, talking).
  • 8 group intent labels (e.g., commuting, eating, discussing).
  • A mix of multi-member groups and single people.

Key Insights and Ideas

  • The One-Way Street: The most significant finding is the lack of mutual reinforcement. While action supervision lifts intent scores, intent supervision never improves action recognition. The "details" help the "whole," but the "whole" is useless for the "details."
  • Groundedness is Mandatory: The model isn't just learning a "script" (e.g., "if people are standing at a bus stop, they must be waiting"). When the authors forced the model to use actions from a different group with the same intent, performance dropped. The gain requires correct, image-grounded actions.
  • Targets vs. Inputs: Interestingly, providing actions as input (hints) in the prompt actually hurt performance. The model learned to take "shortcuts" rather than looking at the image. To get the benefit, the actions must be part of the training target.
  • The Vision Tower Transformation: Through component grafting, the authors found that about 60% of the performance gain is stored in the "vision tower"—the part of the model that encodes the image—rather than just the language reasoning layers.

Key Results

The quantitative results confirm the one-way transfer:

  • Intent Improvement: Writing actions before intent (J-FC) raised the intent macro-F1 score from 30.1 to 31.0. Writing intent first (J-CF) reached 31.2.
  • Action Stagnation: In almost every joint training scenario, action F1 scores stayed within 0.2 points of the action-only model.
  • Label Scarcity: The benefit of action supervision is most pronounced when group intent labels are scarce. With only 25% of intent labels available, the joint model outperformed the intent-only model by a massive 2.6 points.
Figure 3: Intent macro-F1 when intent labels are kept for 25%, 50% or all training groups (seed 42; two draws per budget, averaged). Action labels are always complete.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Scope: The study is limited to JRDB-Social, which represents a specific university environment.
  • Task Difficulty: Social intent is notoriously hard to judge from static frames; even human agreement on the "gold" labels is relatively low (kappa of 0.33).
  • Generalization: The models still struggle with "unseen" locations that weren't in the training set, where the intent-only model sometimes performed better.

Future work might explore whether this one-way transfer holds in other multi-granularity tasks, such as part-of-speech tagging vs. sentiment analysis in text, or object detection vs. scene classification in computer vision. For now, the takeaway for practitioners is simple: if you want your VLM to understand the "why" of a group, first ask it to describe the "what" of the individuals.

Summary updated

Abstract

Social scene datasets label people at two levels: what each person is doing and what the group is there for. Training on both levels at once is often assumed to help each of them. We test this assumption for vision-language models (VLMs) on JRDB-Social, treating person actions as the details and group intent as the whole. We fine-tune Qwen3.5 4B under eight training arms with three seeds, repeat the core arms on Qwen3.5 9B and Qwen3-VL 4B, and judge every arm with a verdict rule fixed before testing. Writing each member's actions before the intent raises intent macro-F1 from 30.1 to 31.0, and writing the intent first reaches 31.2, while action F1 stays within 0.2 points of an action-only model. Under the selected recipe, no arm trained with intent improves actions. The gain requires correct, image-grounded actions: empty action lists or actions borrowed from another group with the same intent lower intent F1. Forced-prefix tests, occlusion, member masking, linear probes and component grafting show that the intent follows the written actions, relies more on people and partly resides in the vision tower. Supplying actions as input instead of as targets hurts both levels.
BibTeX
@misc{gan2026do,
  author = {Chengguang Gan and Zhixi Cai and Yunhao Liang and Hanjun Wei and QingHao Zhang and Shiwen Ni},
  title = {{Do the Details Help the Whole? One-Way Transfer from Person Actions to Group Intent in VLMs on JRDB-Social}},
  year = {2026},
  eprint = {2611.16581},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16581},
}
Figure