Twins: Learn to Predict Unified Representations with Focal Loss
Lists
Only you see this
Abstract
Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.
BibTeX
@inproceedings{gong2026twins,
author = {Kaixiong Gong and Xin Cai and Bin Lin and Hao Wang and Yunlong Lin and Mingzhe Zheng and Bohao Li and Jian-Wei Zhang and Miles Yang and Zhao Zhong and Liefeng Bo and Xiangyu Yue},
title = {{Twins: Learn to Predict Unified Representations with Focal Loss}},
booktitle = {International Conference on Machine Learning},
year = {2026},
url = {https://openreview.net/forum?id=UvWZFnG1Np},
}Loading related content...