HYDRA: Representation Harmonized Tokenization
for Multimodal Generation and Understanding

NeurIPS 2026

Xuerui Qiu^1,2,3, Yutao Cui², Guozhen Zhang^2,4, Junzhe Li^2,5, Yaqi Zhao^2,5, Xiao Zhang²,
Yang Li², Songtao Liu², Miles Yang², Yu Shi³, Zhao Zhong², Liefeng Bo²

¹Institute of Automation, Chinese Academy of Sciences ²Tencent Hunyuan ³Zhongguancun Academy ⁴Nanjing University ⁵Peking University

📄 Paper ⌘ Code 🤗 Models ◌ Training Logs ▤ Results

Figure 1a: HYDRA comparison with unified and task-specific models across multimodal understanding and image generation benchmarks. — **Figure 1a.** HYDRA outperforms unified and task-specific models across diverse multimodal understanding and image generation benchmarks.

Abstract

Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches often employ decoupled encoders, stack representation encoders atop VAEs, or use discrete quantization, disrupting information coherence and creating optimization conflicts. We introduce HYDRA-TOK, a representation-harmonized pure ViT that progressively evolves from a Generation ViT for structure-preserving primitives to a Semantic ViT for semantic encoding. A Generation-Semantic Bottleneck compresses features for robust synthesis and reconstructs them for semantic comprehension. Built on this foundation, HYDRA integrates perception and generation in a single parameter space. HYDRA achieves strong reconstruction, generation, and understanding results: 0.08 rFID under the 448→256 evaluation protocol, 0.86 on GenEval, 86.4 on DPG-Bench, 0.53 on WISE, and an average improvement of about 10 points over previous native UMMs across eight understanding benchmarks.

Figure 2: representation schemes in native unified multimodal models, including the HYDRA representation-harmonized ViT design. — **Figure 2.** Representation schemes in native unified multimodal models. HYDRA uses one ViT backbone and a bottleneck module to harmonize generation and understanding.

Method

HYDRA-TOK restructures a single ViT into a progressive learner. Gen-ViT first preserves the structural primitives needed for high-fidelity synthesis. The Generation-Semantic Bottleneck then compresses these features to filter noise for generation and reconstructs them for downstream semantic learning. Sem-ViT maps the recovered representation into a semantic space. HYDRA uses this coherent visual representation in one shared multimodal backbone, with autoregressive text prediction and rectified-flow image generation heads.

Figure 3: HYDRA-TOK progressive learner and HYDRA dual-head unified architecture. — **Figure 3.** Training process illustration for HYDRA-TOK and HYDRA: progressive visual learning, then a dual-head unified model for text and image generation.

1Representation-harmonized tokenizer. A pure ViT progressive learner reconciles the distinct structural and semantic requirements of generation and understanding.

2Native unified framework. HYDRA processes visual and textual signals in a single parameter space, preserving coherent information flow.

3Mutual improvement through joint training. Generation and understanding co-promote rather than compete, yielding strong results in both directions.

Findings

Identifying the representation and architecture sweet spots

Figures 4 and 5 isolate the two decisions that make the shared representation work. The Generation-Semantic Bottleneck must be compact enough to remove redundant spatial noise, yet expressive enough to preserve reconstruction and semantic information. At the architecture level, the 24 layers must also be divided between structural and semantic processing: 24+0 lacks semantic depth, while imbalanced splits weaken one of the two capabilities. The balanced 12+12 design achieves the strongest reconstruction, understanding, and generation together.

Reconstruction ablation over latent channel dimensions. — **Figure 4a.** Channel dimension: reconstruction.

Understanding and generation ablation over latent channel dimensions. — **Figure 4b.** Channel dimension: understanding and generation.

Reconstruction ablation over Gen-ViT and Sem-ViT layer allocations. — **Figure 5a.** Layer allocation: reconstruction.

Understanding and generation ablation over Gen-ViT and Sem-ViT layer allocations. — **Figure 5b.** Layer allocation: understanding and generation.

Joint training turns two objectives into mutual gains

Figure 6 tracks the learning dynamics of separate and joint objectives. For generation, multimodal supervision stabilizes the latent space and improves both convergence speed and final quality. For understanding, generation initially adds optimization difficulty, but its fine-grained structural constraints become beneficial after the crossover point. The benchmark-level trajectories in Figure 10 show that this trend holds across perception, reasoning, and OCR tasks.

Generation learning curves for joint and generation-only training. — **Figure 6a.** Joint training improves generation efficiency and final quality.

Understanding learning curves for joint and understanding-only training. — **Figure 6b.** Generation constraints ultimately improve understanding.

Performance comparison of joint and understanding-only training across six benchmarks at different training steps. — **Figure 10.** Performance across training steps on general perception (SEED-Bench), reasoning (MMStar, MMMU, MMBench, AI2D), and OCR-related tasks (OCRBench).

Each tokenizer objective contributes to the final balance

Figure 7 isolates the tokenizer training objectives. Teacher initialization supplies a semantic scaffold, bottleneck regularization shapes a stable compact latent space, and self-distillation improves semantic transfer. Their combination gives the best compromise between reconstruction fidelity, generation quality, and multimodal understanding.

Ablation of HYDRA-TOK training objectives. — **Figure 7.** Tokenizer objective ablation: teacher initialization, GSB regularization, and semantic distillation.

Generation and understanding occupy a coherent feature space

Figure 9 visualizes the learned generation and understanding representations. HYDRA forms clearer semantic clusters than the compared unified models, while maintaining alignment between the two branches. This organization supports the continuous information flow that the architecture is designed to preserve.

t-SNE comparison of generation and understanding representations from HYDRA and baseline unified models. — **Figure 9.** Qualitative comparison of t-SNE.

Experimental Results

Click a result category to inspect the complete benchmark table.

Visual reconstruction rFID 0.08 ↓click to collapse

HYDRA-TOK provides a high-fidelity visual foundation while retaining representations useful for multimodal understanding.

Method	Output / evaluation	PSNR ↑	SSIM ↑	rFID ↓
Generative-only Tokenizers
SD-VAE	256² → 256²	23.54	0.68	1.22
SD-VAE 3	256² → 256²	31.29	0.87	0.20
FLUX-VAE	256² → 256²	32.74	0.92	0.18
FLUX-VAE	448² → 448²	32.91	0.92	0.08
Unified Tokenizers
UniTok	256² → 256²	27.28	0.77	0.41
UniFlow	448² → 256²	33.23	0.96	0.26
HYDRA-TOK	448² → 256²	36.39	0.96	0.08
HYDRA-TOK	448² → 448²	31.90	0.90	0.12

ImageNet-1K validation. “448² → 256²” follows the unified-tokenizer protocol: reconstruct at 448², then LANCZOS-resize to 256² before evaluation. “448² → 448²” is native-resolution reconstruction and evaluation.

Multimodal understanding 63.2 / 68.9 Avg. ↑click to expand

Continuous information flow is especially beneficial for detail-sensitive understanding tasks such as OCRBench.

Model	Size	MME-S ↑	AI2D ↑	OCRBench ↑	MMB ↑	RWQA ↑	SEED ↑	MMMU ↑	MMStar ↑	Avg. ↑
Native Unified Multimodal Models
Harmon	1.5B	1476.2	57.0	11.2	67.1	49.8	67.1	38.9	35.3	47.4
Show-o2	1.5B	1742.3	69.0	24.5	67.4	56.5	65.6	37.1	43.4	53.2
HYDRA	1.5B	1877.9	79.4	50.8	76.8	60.7	74.5	39.1	57.0	63.2
Show-o2	7B	1920.5	78.6	32.4	79.3	64.7	69.8	48.9	56.6	62.4
HYDRA	7B	2068.6	85.1	57.7	82.4	64.7	75.5	49.4	62.3	68.9

Text-to-image generation GenEval 0.86 · DPG 86.40 · WISE 0.53click to expand

Detailed results on GenEval, DPG-Bench, and WISE. Higher is better for every metric; DPG-Bench reports Overall only.

Model	Size	GenEval							DPG-Bench	WISE
Model	Size	Single Obj. ↑	Two Obj. ↑	Count ↑	Colors ↑	Pos. ↑	Color Attr. ↑	Overall ↑	Overall ↑	Culture ↑	Time ↑	Space ↑	Biology ↑	Physics ↑	Chemistry ↑	Overall ↑
Native Unified Multimodal Models
Harmon	1.5B	0.99	0.86	0.66	0.85	0.74	0.48	0.76	—	0.38	0.48	0.52	0.37	0.44	0.29	0.41
Show-o2	1.5B	0.99	0.86	0.55	0.86	0.46	0.63	0.73	85.02	0.33	0.41	0.53	0.35	0.49	0.31	0.39
HYDRA	1.5B	1.00	0.95	0.75	0.91	0.76	0.77	0.86	85.51	0.49	0.47	0.64	0.42	0.59	0.34	0.49
Show-o2	7B	1.00	0.87	0.58	0.92	0.52	0.62	0.76	86.14	0.40	0.45	0.58	0.39	0.53	0.34	0.44
HYDRA	7B	1.00	0.97	0.68	0.91	0.81	0.80	0.86	86.40	0.52	0.53	0.68	0.47	0.58	0.38	0.53

Qualitative Results

HYDRA text-to-image generation examples across diverse subjects, objects, text, and landscapes. — HYDRA text-to-image generation examples.

HYDRA text-to-image generation examples. — Text-to-image generation results.

Reconstruction comparison between HYDRA-TOK and baselines. — Reconstruction comparison with visual tokenization baselines.

HYDRA multimodal understanding examples. — Multimodal understanding examples.

Qualitative generation comparison of HYDRA and other methods. — Qualitative comparison on image generation prompts.

Visitors around the world

Explore where HYDRA's readers come from.

World map with flags marking the countries of HYDRA visitors — Global visitor map

The 20 most active visitor countries, with a visit count beside each flag — Visits by country

Shared across the project page and training logs. A visitor is counted once per 24 hours; updates may take a few minutes.

View all countries & visitor history

Country estimates and counts provided by Flag Counter.

Open the visitor report to view the world map, visitor totals, and country counts.

Citation

Read the paper: arXiv:2603.15228

Please cite HYDRA using the following BibTeX entry.

@article{qiu2026hydra,
  title={{HYDRA}: Representation Harmonized Tokenization for Multimodal Generation and Understanding},
  author={Qiu, Xuerui and Cui, Yutao and Zhang, Guozhen and Li, Junzhe and Zhao, Yaqi and Zhang, Xiao and Li, Yang and Liu, Songtao and Yang, Miles and Shi, Yu and Zhong, Zhao and Bo, Liefeng},
  journal={arXiv preprint arXiv:2603.15228},
  year={2026}
}