Where Temporal Multi-Task Conflict Lives: A Frequency View of Video Gradients

When one network is trained on several objectives at once (multi-task learning), the tasks can help each other or fight over the shared features. Methods that manage this fight have been checked on still images: CIFAR-10, CelebA, and single-frame scene understanding on NYUv2. In video, the tasks that matter most differ in temporal granularity. One task labels the whole clip, another labels every frame, and a third finds where each action starts and ends. This proposal asks whether these tasks conflict, and where.

Our hypothesis is that the conflict sits in a specific place: in the time-frequency content of the shared features. A clip-level head averages over time, so it pushes the features to vary slowly, like a low-pass filter. A boundary head needs sharp frame-to-frame change. If that is right, then splitting the shared features into slow and fast temporal bands and resolving disagreements separately within each band should beat both naive loss summation and whole-space gradient methods. We test this on dense action detection, where strong single-task numbers already exist.

Core Claim

The claim has two parts, and each can fail on its own terms. The first part is diagnostic. When a clip-level classification task and a frame- or boundary-level task share a temporal encoder, their gradients agree in the slow temporal band and disagree mostly in the fast band. The second part is about method. Handling the conflict separately in each band gives better dense temporal detection than summing losses, than applying RotoGrad to the flattened features, and than simply cutting the clip-level head off from the shared features.

If the measured conflict is spread evenly across bands, the first claim is false and the method has no reason to exist. We would then report that negative result about how image-validated multi-task tools carry over to video.

The idea borrows a finding from static vision. FMViT, which mixes high- and low-frequency spatial components, reports that high spatial frequencies are what dense tasks need for sharp boundaries, while global classification is carried by low frequencies. We move that observation from space to time and turn it from an architecture choice into a statement about gradients. FMViT itself only fine-tunes each task separately and never trains them jointly, so it says nothing about conflict.

Positioning Against Prior Work

Three lines of work motivate the gap.

RotoGrad handles conflicting task gradients in two ways. It reweights gradient magnitudes toward tasks that are learning slowly, and it gives each task $k$ a learned rotation $R_k$ of the shared features $z$, so that task $k$ reads $r_k = R_k z$. All of its evidence is on static data, and the paper notes that full rotations become costly in high dimensions. RotoGrad supplies our magnitude-balancing component and is our strongest generic competitor. Our point of difference is that we impose the structure of time on where the conflict is resolved, rather than learning an unstructured rotation over a space of size $T \times D$.

DeMT is a multi-task decoder for dense image tasks that uses task-specific queries. It reports a 3.36% average per-task gain ($\Delta_m$) over single-task models on NYUD-v2, compared with 0.31% for a plainly shared design. This tells us that task-aware separation matters, but it has no time axis.

The dense video detectors constrain our design.

An Effective-Efficient Approach for Dense Multi-Label Action Detection uses two pieces:

  • a label autoencoder ("assistant branch") whose decoder weights are copied into the classifier at train time and discarded at inference;
  • a non-hierarchical transformer.

Reverting to a hierarchical, downsampling design costs 1.4% mAP on Charades. That rules out temporal pyramids for our encoder, so we keep full temporal resolution. Removing the assistant branch costs 0.5%. We ask whether that 0.5% is really a symptom of unmanaged interference that gradient handling can recover.

MLAD models class co-occurrence and temporal dependencies with attention. PointTAD represents each action instance by learnable temporal query points and supplies our boundary head. C2F-TCN already pairs frame-level segmentation with a clip-level activity label on Breakfast. That pairing gives us a clip-level task whose label is not derived from the frame labels.

One paper appears to contradict us. Dense Video Object Captioning reports that disjoint detection, tracking and captioning objectives combine well with no gradient management. We read this as evidence that conflict depends on the task pair, not as a refutation. Its tasks attach to different output structures (boxes, tubes, text). Ours all compete for the same per-frame features. Our measurement step is designed to tell which situation holds.

We set aside language-model heads for timestamps. Both STVG-R1 and TRAM document numeric and temporal errors in that setting. We therefore use dedicated detection heads.

Research Program

We fix a shared temporal encoder: a non-hierarchical transformer over pre-extracted I3D clip features, matching the feature protocol of the baselines. It produces $z \in \mathbb{R}^{T \times D}$, with $T$ time steps and $D$ channels. Three heads read $z$:

  • a clip-level multi-label classifier that pools over time;
  • a per-frame multi-label classifier;
  • a PointTAD-style instance head with query points and matching.

We split the gradient that each task sends to $z$ into temporal frequency bands. Let $P_b$ project onto band $b$ of a discrete cosine transform along the time axis. The bands are fixed, octave-spaced groups of coefficients. Task $k$'s gradient in band $b$ is

$$g_k^{(b)} = P_b \, \nabla_z \mathcal{L}_k ,$$

where $\mathcal{L}_k$ is task $k$'s loss. We then measure the per-band agreement

$$c_{jk}^{(b)} = \cos\big(g_j^{(b)}, g_k^{(b)}\big)$$

for every task pair $(j,k)$ throughout training. This is the diagnostic.

The method works one band at a time:

  1. Equalize magnitudes across tasks, following RotoGrad's slow-task reweighting.
  2. Where $c_{jk}^{(b)} < 0$, remove from each gradient its component along the other.
  3. Sum the bands back together with the inverse transform, and backpropagate into the encoder.

The added cost is one transform along the time axis per step, which is small next to attention over $T$.

We will run the diagnostic first on all three benchmarks. The method is worth building only if the conflict concentrates in the high bands.

Baselines

The naive reference point is the frame-level detector trained alone, with the same encoder, features and schedule. Every multi-task variant must beat it, or multi-task training is pointless here.

The strongest published system is the assistant-branch method of the dense action detection paper. It reports 26.5% per-frame mAP on Charades and 44.6% on MultiTHUMOS. We will reproduce it under our feature pipeline and report both the reproduced and the published number. MLAD reports 23.7% f-mAP on Charades and 51.5% on MultiTHUMOS. Its MultiTHUMOS figure exceeds the 44.6% "state of the art" claimed by the newer paper, which suggests the two used different evaluation protocols. We will rerun both under one evaluation script rather than compare published numbers. For instance-level detection, PointTAD reports 23.5% detection-mAP and 41.2% segmentation-mAP on MultiTHUMOS. For Breakfast, C2F-TCN reports only relative gains (+3.3 F1@50, +5.9 frame accuracy from feature augmentation). We found no absolute Breakfast number on Lacuna, so we will reproduce one.

The multi-task competitors share our encoder and heads and differ only in how gradients reach $z$:

  • a fixed weighted sum of losses, with weights tuned by grid search;
  • RotoGrad applied to the flattened $T \times D$ features, using its subspace approximation;
  • a magnitude-only variant of our method;
  • a direction-only variant of our method.

The last two ablations isolate which part of the per-band handling matters.

Every method gets the same number of hyperparameter trials, the same epochs, and three seeds. Beating a baseline means a mean improvement larger than two seed standard deviations on per-frame mAP and detection-mAP. We also require that the clip-level metric does not drop.

Why not just…?

Why not just sum the losses with tuned weights? RotoGrad and DeMT both show that on images, a plain sum lets one task dominate the shared features. DeMT's shared design gains only 0.31% over single-task models. Still, video tasks built from the same annotations may conflict less than image tasks. This baseline therefore gets the same tuning budget as our method, and the per-band diagnostic will show whether there was any conflict for weighting to fix.

Why not just apply RotoGrad as is? RotoGrad handles both magnitude and direction. Over flattened $T \times D$ features, though, its rotation has no notion of time and must be approximated in a subspace. If RotoGrad matches us, the frequency structure adds nothing, and we will say so. The magnitude-only and direction-only variants show which component carries any gain.

Why not just detach or low-pass the clip-level head? This is the most serious threat. If the clip head only harms the fast band, two simpler fixes may be enough: stop its gradient into $z$ entirely, or wire it to read only the low-pass part of $z$. Both are architectural, with no gradient surgery. We run both. If low-pass routing equals per-band homogenization, we will recommend routing as the simpler method and keep the diagnostic as the contribution. If detaching equals both, clip-level supervision adds nothing to dense video detection, which is itself worth knowing for C2F-TCN-style designs.

Why not just keep the assistant branch? It costs nothing at inference and is worth 0.5% on Charades. We test whether per-band handling, without the assistant branch, recovers that 0.5%. We also test whether the two gains add up.

Proposed Experiments

The first decisive check is the diagnostic on Charades, MultiTHUMOS and Breakfast. We plot $c_{jk}^{(b)}$ by band over the course of training. The prediction is positive or near-zero agreement in the lowest band and negative agreement in the highest bands for the clip-versus-frame and clip-versus-boundary pairs. A flat profile across bands falsifies the core claim.

Breakfast is especially telling. Its clip-level activity label is a different label from the frame-level actions. On Charades and MultiTHUMOS, the clip labels can be derived from the frame labels, so Breakfast separates real task diversity from relabelling.

The second check is performance under the matched protocol above. We report:

  • per-frame mAP on Charades and MultiTHUMOS;
  • detection-mAP and segmentation-mAP on MultiTHUMOS;
  • F1@50 and frame accuracy on Breakfast.

The third check measures where the gain comes from. If our story is right, the improvement should appear mostly in boundary-sensitive measures, not in average frame accuracy:

  • the instance-level detection-mAP;
  • the action-conditional recall metric used by the assistant-branch paper, which measures recall of co-occurring actions given that one action is present.

Finally, we sweep the band cutoffs. This checks that the result does not hinge on one particular way of splitting the frequencies.