Agnes-Video-2.0-MoE
Agnes-Video-2.0-MoE is an audio-video generation model based on DaVinci-MagiHuman. It accepts text, video, and audio inputs and generates video and audio. With TIRx kernels, a 5-second 480×272 clip at 32 denoising steps now takes 13.641 s on an NVIDIA B200.
Model Description
Agnes-Video-2.0-MoE uses a 40-layer, single-stream multimodal DiT. The shared feed-forward networks in layers 4–35 are converted to three-expert SparseMoE blocks with Top-1 routing. checkpoint_parts/ is the EENP p60 checkpoint: modality-aware neuron pruning across video, audio, and text, with elastic Quality and Fast widths.
| Attribute | Description |
|---|---|
| Base model | DaVinci-MagiHuman |
| Model type | Audio-video generation model |
| Input modalities | Text, video, audio |
| Output modalities | Video, audio |
| DiT modification | Layers 4–35; three experts with Top-1 routing |
| Uploaded checkpoint | EENP p60 in checkpoint_parts/ |
| Unpruned SparseMoE parameters (stored / active) | 28.721B / 15.299B |
| Elastic inference | Quality / Fast widths |
The uploaded checkpoint uses two nested widths. Quality retains 65% of intermediate neurons, and Fast retains 40% as a high-score subset of Quality. A 32-step run uses 8 Quality steps and 24 Fast steps. These percentages describe routed FFN neuron widths, not the fraction of total model parameters removed. The parameter counts above are the unpruned SparseMoE model. Separately obtained text encoder and VAE weights are not included.
Video Demos
The examples below are text-conditioned generations. Click a preview to open its MP4 video.
| Orange cat licking its paw.  |
Brown-and-white dog playing on a sunny lawn.  |
Monkey peeking over a red apple.  |
| White rabbit in a terracotta flowerpot.  |
Golden sunrise over snow-covered peaks.  |
Horses grazing beside a river on grassland.  |
Results
Open-Generator Comparison
The comparison uses 32 Koala clips. The “Ours” row is the unpruned SparseMoE checkpoint (28.7B stored / 15.3B active parameters). The weights in checkpoint_parts/ are the EENP p60 checkpoint.
Stage Results and Quality–Speed Trade-off
checkpoint_parts/ is this EENP 60% checkpoint. Relative to unpruned SparseMoE, it reduces DiT denoising latency from 26.83 s to 20.92 s (22.0% lower; 1.28× speedup). VBench 2.0 changes from 0.519 to 0.479. On the paired diagnostic set, FVD changes from 64.267 to 121.012 and CLIP-Sim from 0.9186 to 0.8227, reflecting the quality–speed trade-off.
| Metric | Evaluation setup |
|---|---|
| VBench 2.0 | 448×256; 4 seconds; 32 denoising steps; guidance 2 |
| DiT latency | NVIDIA H100; FP32; batch size 1; mean over 128 five-second generations after one warm-up; excludes VAE decoding |
| Paired diagnostics (FVD / CLIP-Sim) | Fixed 32-clip set; 448×256; 5 seconds; 32 denoising steps; guidance 2; seed 42 |
Inference
Set the component paths in the inference configuration, then run:
export OMNIVAE_CODE_ROOT=/path/to/OmniVAE/vae
export OMNIVAE_RELEASE_ROOT=/path/to/OmniVAE/release
cat checkpoint_parts/ckpt_best.pt.part-* > ckpt_eenp_p60.pt
TIRX_ELASTIC=1 bash inference_code/infer.sh \
--config-load-path /path/to/config_moe32_omnivae_init.json \
--ckpt_dir ckpt_eenp_p60.pt \
--generate \
--prompt "A small sailboat crossing a calm lake at sunset." \
--seconds 5 \
--num_inference_steps 32 \
--cfg_number 2 \
--save_path_prefix outputs/demo
TIRx kernels
checkpoint_parts/ is the EENP p60 checkpoint, and denoising uses TIRx kernels. On an NVIDIA B200, one 5-second 480×272 clip at 32 steps now takes 13.641 s (denoising only). That is 4.150 s faster than the 17.791 s PyTorch path, a 23.33% reduction.
TIRX_ELASTIC=1 is the path for these weights. It needs tirx_kernels on Blackwell (sm_100).
Model Components
| Component | Source |
|---|---|
| Text encoder | T5Gemma 9B |
| Audio encoder | OmniVAE |
| Video VAE | Wan2.2-TI2V-5B |
License and Third-Party Dependencies
Agnes-Video-2.0-MoE checkpoints contain only Agnes-Video-2.0-MoE DiT/MoE weights. No pretrained base-model weights or third-party model weights are included or redistributed.
Agnes-Video-2.0-MoE code and model weights are provided under the Apache License 2.0, subject to applicable upstream license requirements. See LICENSE and NOTICE.
Agnes-Video-2.0-MoE is developed based on DaVinci-MagiHuman. We acknowledge the original project and retain applicable license notices and attribution.
For inference, users must separately obtain required base-model and auxiliary-model weights from their respective official repositories.
Google T5Gemma is an external text encoder governed by the Gemma Terms of Use, not Apache 2.0. Users must obtain T5Gemma independently and comply with its applicable terms.
Third-party models and dependencies remain subject to their respective licenses. The Apache License 2.0 for Agnes-Video-2.0-MoE does not extend to these external components.
References: