Revisiting Multi-Task Learning with ROCK: a Deep Residual Auxiliary Block for Visual Detection

Taylor Mordan, Nicolas Thome, Gilles Hénaff, Matthieu Cord

Deep learning models, particularly Convolutional Neural Networks (ConvNets), are notoriously data-hungry. In specialized domains like medical imaging or military applications, collecting millions of images is often impossible. However, it is sometimes easier to get additional types of annotations (auxiliary data) for the few images you do have.

To address this, researchers introduce ROCK (Residual Auxiliary Block), a novel architectural component that boosts the performance of a primary task (like object detection) by leveraging auxiliary tasks (like depth estimation) during training. By using a residual connection to inject auxiliary features directly into the main network path, ROCK significantly outperforms standard multi-task learning approaches on small datasets without requiring auxiliary data at inference time.

The Problem: "Flat" vs. "Primary" Multi-Task Learning

Standard Multi-Task Learning (MTL) treats all tasks as equals, an approach often called "Flat MTL." While this helps regularize the network, it isn't optimized for scenarios where you only care about one "primary" task.

The authors propose focusing on Primary MTL, where the sole goal is to maximize the performance of a specific task by using others as "privileged information" available only during training.

Methods: The ROCK Architecture

The core contribution is the ROCK block, a generic multi-modal fusion block that can be inserted into existing architectures (like SSD for object detection). It consists of four main components:

  1. Encoder: Extracts task-specific features from the base network's feature maps.
  2. Predictor: Generates outputs for auxiliary tasks (e.g., a depth map or scene class). Crucially, this component contains no learnable parameters—only pooling operations. This forces the Encoder to learn highly representative features because the Predictor cannot "fix" poor inputs with its own weights.
  3. Decoder: Projects the encoder's task-specific features back into the original feature space.
  4. Fusion: Uses a residual connection to add these auxiliary features back into the primary task's path.
Detailed architecture of the ROCK block

By injecting auxiliary information directly into the forward pass via the residual connection, the primary task is explicitly influenced by the geometric and semantic context provided by the auxiliary tasks.

Key Insights

  • The "No-Parameter" Predictor Trick: By making the auxiliary predictor a simple pooling operation, the authors ensure that the features learned in the encoder are "ready-to-use." This forces the network to embed the auxiliary knowledge (like the shape of a chair or the depth of a room) directly into the features that get merged back into the main detection path.
  • Breaking Symmetry: Unlike flat MTL, where tasks share a backbone but branch off equally, ROCK's residual connection creates an asymmetrical relationship. The auxiliary tasks "serve" the primary task by refining its feature representations.
  • Privileged Information: ROCK acts as a bridge for "Learning Using Privileged Information" (LUPI). You use depth and surface normals to train a better RGB detector, but at test time, you only need the RGB image. The "knowledge" of depth is already baked into the refined weights.

Key Results

The model was heavily evaluated on the NYUv2 dataset (1,449 indoor images) with object detection as the primary task, and scene classification, depth prediction, and surface normal estimation as auxiliary tasks.

Ablation study showing ROCK's performance gains over baselines
  • State-of-the-Art Performance: ROCK achieved 37.6 mAP (mean Average Precision), a significant jump from the 31.2 mAP baseline. It outperformed previous "Modality Hallucination" methods by over 3 points.
  • Synthetic-to-Real Transfer: Pre-training on a large-scale synthetic dataset (MLT) and fine-tuning on NYUv2 led to a massive performance boost, reaching an impressive 46.8 mAP.
  • Data Efficiency: The authors found that training ROCK on only 70% of the training data yielded results comparable to the baseline trained on 100% of the data. This proves that auxiliary tasks can effectively compensate for a lack of raw image volume.
  • Low Overhead: Despite the performance boost, ROCK only adds about 17% more parameters and is only 7% slower during inference compared to a standard SSD model.

Limitations and Future Work

While highly effective, the current version of ROCK uses simple element-wise addition for fusion. The authors suggest that more complex methods, like gated mechanisms or bilinear fusion, could potentially yield even better results. Additionally, while the paper demonstrates success with depth, normals, and scene labels, it does not provide a formal framework for choosing which auxiliary tasks are most beneficial for a given primary task. Future testing on diverse outdoor or non-natural image domains (like satellite or medical imagery) will further strengthen the case for ROCK's universality.

Summary updated

Abstract

Multi-Task Learning (MTL) is appealing for deep learning regularization. In this paper, we tackle a specific MTL context denoted as primary MTL, where the ultimate goal is to improve the performance of a given primary task by leveraging several other auxiliary tasks. Our main methodological contribution is to introduce ROCK, a new generic multi-modal fusion block for deep learning tailored to the primary MTL context. ROCK architecture is based on a residual connection, which makes forward prediction explicitly impacted by the intermediate auxiliary representations. The auxiliary predictor's architecture is also specifically designed to our primary MTL context, by incorporating intensive pooling operators for maximizing complementarity of intermediate representations. Extensive experiments on NYUv2 dataset (object detection with scene classification, depth prediction, and surface normal estimation as auxiliary tasks) validate the relevance of the approach and its superiority to flat MTL approaches. Our method outperforms state-of-the-art object detection models on NYUv2 dataset by a large margin, and is also able to handle large-scale heterogeneous inputs (real and synthetic images) with missing annotation modalities.
BibTeX
@inproceedings{mordan2018revisiting,
  author = {Taylor Mordan and Nicolas Thome and Gilles Hénaff and Matthieu Cord},
  title = {{Revisiting Multi-Task Learning with ROCK: a Deep Residual Auxiliary Block for Visual Detection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=zcoJ0QZ4McR},
}
Figure