TL;DR
Lumina-OmniLV (OmniLV) is a unified Diffusion Transformer (DiT) framework capable of executing over 100 low-level vision sub-tasks across restoration, enhancement, dense prediction, and stylization via both text instructions and visual prompts. By decoupling text and visual guidance and co-training the DiT base model with early-layer condition injection, OmniLV generates high-fidelity outputs at arbitrary resolutions up to 1K without suffering from task ambiguity.
What Problem Does This Paper Address?
While generalist multimodal foundation models (such as GPT-4V, InternVL, and OmniGen) have transformed high-level understanding and creative generation, low-level vision remains fragmented. Prior solutions have fallen into a few restrictive camps:
- Task-specific models (e.g., deblurring or denoising networks) that require dedicated architectures and separate retraining for every task.
- "All-in-one" restoration models that handle composite degradations but cannot generalize across domains to structural tasks like edge detection or depth estimation.
- Visual-prompt-only frameworks (e.g., PromptGIP, GenLV) that require awkward example-image pairs and lack flexible natural-language control.
- Fixed-resolution models unable to handle flexible aspect ratios or modern high-resolution imagery.
Unifying low-level vision is uniquely tricky: unlike high-level tasks that rely on semantic abstraction, low-level tasks require extreme pixel-level fidelity, fine texture preservation, and operation across completely different output spaces (e.g., continuous RGB images vs. discrete surface normal maps).
Methods
OmniLV adapts a flow-matching Diffusion Transformer (Lumina-Next) for general multi-task low-level vision:
- Decoupled Multimodal Conditioning: Instead of passing text instructions and visual prompt images through a shared Multimodal Large Language Model (MLLM), OmniLV decouples them. Natural-language instructions and descriptions are processed via a language model (Gemma-2B), while visual prompts are independently encoded with a visual VAE.
- In-Context Visual Prompting via Projection-Addition: For tasks that are difficult to articulate in text (such as artistic stylization), users can supply visual input–output exemplar pairs. Instead of token concatenation, OmniLV aligns prompt features into the latent space using lightweight linear projectors and adds them directly to the target image latent.
- Co-Trained Condition Adapter: Rather than freezing the diffusion backbone in a ControlNet fashion, OmniLV jointly fine-tunes the base DiT and a 12-layer transformer condition adapter, injecting condition image features into the first half of the network layers for early guidance.
- Three-Stage Progressive Training:
- Single-image tasks at $512 \times 512$ resolution.
- Integration of visual in-context learning tasks at $512 \times 512$.
- Full multi-task training scaled up to $1024 \times 1024$ (1K) resolution on 16 NVIDIA A100 GPUs using a Conditional Flow Matching (CFM) loss.
Text Instruction ──> Gemma-2B ────┐
├──> Lumina-Next DiT (Co-trained) ──> VAE Decoder ──> Output
Condition Image ──> VAE + Adapter ┘ ▲
│ (Projection-Addition)
Visual Exemplars ──> VAE + Projectors ──────┘
Datasets
- OmniLV Training Dataset: A multi-task corpus of 40 million instances covering over 100 sub-tasks across four primary domains:
- Image Restoration (49.7%): 23 major tasks / 45 sub-tasks (blur, Gaussian/Poisson noise, compression, rain, snow, haze, face SR, inpainting, reflection, flare, watermark removal, etc.).
- Image Enhancement (21.9%): 14 major tasks / 25 sub-tasks (exposure, low-light, brightening, darkening, contrast, saturation, retouching, bokeh, white balance).
- Weak-Semantic Dense Prediction (17.8%): 10 tasks (relative depth estimation, surface normals, Canny edge, HED edge, segmentation, saliency).
- Image Stylization (10.6%): 20 tasks (pencil drawing, local Laplacian filtering, artistic filters).
- OmniLV-Test (OLV-T): A newly introduced evaluation benchmark built from DIV2K-val, containing 44 task-specific test splits with 100 images each (4,400 test images at 1K resolution).
- Standard Public Benchmarks: RealBlur-J, SIDD, Rain1400, Snow100K-L, DIV2K, CelebA, LOL-v2-Real, MIT5K, and NYU-v2.
Key Insights and Ideas
- Shared MLLMs cause task interference: Passing both text prompts and visual exemplars through a unified MLLM (e.g., Qwen-VL 2.5) causes visual tokens to overpower text instructions in the shared representation space. This leads to severe task confusion (e.g., generating surface normals when prompted for depth). Decoupling text (LLM) and image exemplars (VAE) completely resolves this misalignment.
- High-semantic generative tasks hurt pixel fidelity: Integrating text-to-image generation or semantic image editing into a low-level vision model actively hurts performance on restoration tasks. High-level generation emphasizes conceptual coherence and abstract hallucination, directly conflicting with the strict pixel-accuracy needed for restoration.
- Multi-task synergy exists within low-level vision: Jointly training on restoration and enhancement tasks improves restoration metrics over training on restoration alone, demonstrating positive cross-task transfer.
- Text beats visual cues during conflict: When presented with deliberately conflicting prompts (e.g., an in-context exemplar showing depth estimation paired with a text instruction saying "Estimate normal map"), OmniLV reliably adheres to the textual instruction.
Key Results
- Restoration & Enhancement: OmniLV consistently outperforms prior unified and prompt-based models (such as DA-CLIP, PromptGIP, GenLV, PixWizard, and PromptFix) across PSNR and perceptual MUSIQ metrics (e.g., achieving 23.21 dB PSNR / 69.78 MUSIQ on Gaussian denoising and 22.58 dB PSNR / 70.54 MUSIQ on brightening).
- Competitive with Specialized Models: On Blind Image Restoration (BIR) on DIV2K and face restoration on CelebA, OmniLV achieves 22.36 dB / 69.55 MUSIQ and 25.04 dB / 70.70 MUSIQ, rivaling or nearing dedicated models like DiffBIR (22.77 dB / 67.01) and GFPGAN (25.80 dB / 69.76).
- Instruction Following: On dense prediction tasks (Canny, Depth, HED), OmniLV achieves a 100% task execution success rate, outperforming generalist models like OmniGen (75–80%).
- Human Preference: In a pairwise Elo evaluation with 37,500 human votes, OmniLV scored an Elo rating of 1,475, decisively beating PixWizard (1,310) and PromptFix (anchored at 1,000).
Limitations and Future Work
- Performance Gap with Task-Specific SOTA: While OmniLV establishes strong generalist performance across 100+ tasks, specialized standalone models tailored to a single task (such as Depth Anything for relative depth estimation) still hold a slight edge in quantitative metrics.
- Future Work: The authors highlight the need for refined model sub-components and more granular conditioning strategies to bridge the performance gap with task-specific models without losing generalist multi-task capabilities.
Summary updated