Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Vision

Yuandong Pu, Le Zhuo, Kaiwen Zhu, Liangbin Xie, Wenlong Zhang, Xiangyu Chen, Peng Gao, Yu Qiao, Chao Dong, Yihao LiuSubmission to ICLR 2026 · Rejected

TL;DR

Lumina-OmniLV (OmniLV) is a unified Diffusion Transformer (DiT) framework capable of executing over 100 low-level vision sub-tasks across restoration, enhancement, dense prediction, and stylization via both text instructions and visual prompts. By decoupling text and visual guidance and co-training the DiT base model with early-layer condition injection, OmniLV generates high-fidelity outputs at arbitrary resolutions up to 1K without suffering from task ambiguity.

What Problem Does This Paper Address?

While generalist multimodal foundation models (such as GPT-4V, InternVL, and OmniGen) have transformed high-level understanding and creative generation, low-level vision remains fragmented. Prior solutions have fallen into a few restrictive camps:

  1. Task-specific models (e.g., deblurring or denoising networks) that require dedicated architectures and separate retraining for every task.
  2. "All-in-one" restoration models that handle composite degradations but cannot generalize across domains to structural tasks like edge detection or depth estimation.
  3. Visual-prompt-only frameworks (e.g., PromptGIP, GenLV) that require awkward example-image pairs and lack flexible natural-language control.
  4. Fixed-resolution models unable to handle flexible aspect ratios or modern high-resolution imagery.

Unifying low-level vision is uniquely tricky: unlike high-level tasks that rely on semantic abstraction, low-level tasks require extreme pixel-level fidelity, fine texture preservation, and operation across completely different output spaces (e.g., continuous RGB images vs. discrete surface normal maps).

Methods

OmniLV adapts a flow-matching Diffusion Transformer (Lumina-Next) for general multi-task low-level vision:

  • Decoupled Multimodal Conditioning: Instead of passing text instructions and visual prompt images through a shared Multimodal Large Language Model (MLLM), OmniLV decouples them. Natural-language instructions and descriptions are processed via a language model (Gemma-2B), while visual prompts are independently encoded with a visual VAE.
  • In-Context Visual Prompting via Projection-Addition: For tasks that are difficult to articulate in text (such as artistic stylization), users can supply visual input–output exemplar pairs. Instead of token concatenation, OmniLV aligns prompt features into the latent space using lightweight linear projectors and adds them directly to the target image latent.
  • Co-Trained Condition Adapter: Rather than freezing the diffusion backbone in a ControlNet fashion, OmniLV jointly fine-tunes the base DiT and a 12-layer transformer condition adapter, injecting condition image features into the first half of the network layers for early guidance.
  • Three-Stage Progressive Training:
  • Single-image tasks at $512 \times 512$ resolution.
  • Integration of visual in-context learning tasks at $512 \times 512$.
  • Full multi-task training scaled up to $1024 \times 1024$ (1K) resolution on 16 NVIDIA A100 GPUs using a Conditional Flow Matching (CFM) loss.
Text Instruction ──> Gemma-2B ────┐
                                   ├──> Lumina-Next DiT (Co-trained) ──> VAE Decoder ──> Output
Condition Image  ──> VAE + Adapter ┘        ▲
                                            │ (Projection-Addition)
Visual Exemplars ──> VAE + Projectors ──────┘

Datasets

  • OmniLV Training Dataset: A multi-task corpus of 40 million instances covering over 100 sub-tasks across four primary domains:
  • Image Restoration (49.7%): 23 major tasks / 45 sub-tasks (blur, Gaussian/Poisson noise, compression, rain, snow, haze, face SR, inpainting, reflection, flare, watermark removal, etc.).
  • Image Enhancement (21.9%): 14 major tasks / 25 sub-tasks (exposure, low-light, brightening, darkening, contrast, saturation, retouching, bokeh, white balance).
  • Weak-Semantic Dense Prediction (17.8%): 10 tasks (relative depth estimation, surface normals, Canny edge, HED edge, segmentation, saliency).
  • Image Stylization (10.6%): 20 tasks (pencil drawing, local Laplacian filtering, artistic filters).
  • OmniLV-Test (OLV-T): A newly introduced evaluation benchmark built from DIV2K-val, containing 44 task-specific test splits with 100 images each (4,400 test images at 1K resolution).
  • Standard Public Benchmarks: RealBlur-J, SIDD, Rain1400, Snow100K-L, DIV2K, CelebA, LOL-v2-Real, MIT5K, and NYU-v2.

Key Insights and Ideas

  • Shared MLLMs cause task interference: Passing both text prompts and visual exemplars through a unified MLLM (e.g., Qwen-VL 2.5) causes visual tokens to overpower text instructions in the shared representation space. This leads to severe task confusion (e.g., generating surface normals when prompted for depth). Decoupling text (LLM) and image exemplars (VAE) completely resolves this misalignment.
  • High-semantic generative tasks hurt pixel fidelity: Integrating text-to-image generation or semantic image editing into a low-level vision model actively hurts performance on restoration tasks. High-level generation emphasizes conceptual coherence and abstract hallucination, directly conflicting with the strict pixel-accuracy needed for restoration.
  • Multi-task synergy exists within low-level vision: Jointly training on restoration and enhancement tasks improves restoration metrics over training on restoration alone, demonstrating positive cross-task transfer.
  • Text beats visual cues during conflict: When presented with deliberately conflicting prompts (e.g., an in-context exemplar showing depth estimation paired with a text instruction saying "Estimate normal map"), OmniLV reliably adheres to the textual instruction.

Key Results

  • Restoration & Enhancement: OmniLV consistently outperforms prior unified and prompt-based models (such as DA-CLIP, PromptGIP, GenLV, PixWizard, and PromptFix) across PSNR and perceptual MUSIQ metrics (e.g., achieving 23.21 dB PSNR / 69.78 MUSIQ on Gaussian denoising and 22.58 dB PSNR / 70.54 MUSIQ on brightening).
  • Competitive with Specialized Models: On Blind Image Restoration (BIR) on DIV2K and face restoration on CelebA, OmniLV achieves 22.36 dB / 69.55 MUSIQ and 25.04 dB / 70.70 MUSIQ, rivaling or nearing dedicated models like DiffBIR (22.77 dB / 67.01) and GFPGAN (25.80 dB / 69.76).
  • Instruction Following: On dense prediction tasks (Canny, Depth, HED), OmniLV achieves a 100% task execution success rate, outperforming generalist models like OmniGen (75–80%).
  • Human Preference: In a pairwise Elo evaluation with 37,500 human votes, OmniLV scored an Elo rating of 1,475, decisively beating PixWizard (1,310) and PromptFix (anchored at 1,000).

Limitations and Future Work

  • Performance Gap with Task-Specific SOTA: While OmniLV establishes strong generalist performance across 100+ tasks, specialized standalone models tailored to a single task (such as Depth Anything for relative depth estimation) still hold a slight edge in quantitative metrics.
  • Future Work: The authors highlight the need for refined model sub-components and more granular conditioning strategies to bridge the performance gap with task-specific models without losing generalist multi-task capabilities.

Summary updated

Abstract

We present Lunima-OmniLV (abbreviated as OmniLV), a universal multimodal multi-task framework for low-level vision that addresses over 100 sub-tasks across four major categories, including image restoration, image enhancement, weak-semantic dense prediction, and stylization. OmniLV leverages both textual and visual prompts to offer flexible, user-friendly interactions. Built on Diffusion Transformer (DiT)-based generative priors, our framework supports arbitrary resolutions — achieving optimal performance at 1K resolution — while preserving fine-grained details and high fidelity. Through extensive experiments, we demonstrate that separately encoding text and visual instructions, combined with co-training using shallow feature control, is essential to mitigate task ambiguity and enhance multi-task generalization. Our findings also reveal that integrating high-level generative tasks into low-level vision models can compromise detail-sensitive restoration. These insights pave the way for more robust and generalizable low-level vision systems.
BibTeX
@misc{pu2025lumina,
  author = {Yuandong Pu and Le Zhuo and Kaiwen Zhu and Liangbin Xie and Wenlong Zhang and Xiangyu Chen and Peng Gao and Yu Qiao and Chao Dong and Yihao Liu},
  title = {{Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Vision}},
  year = {2025},
  eprint = {2504.04903},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.04903},
}

Loading related content...

Figure