Qwen3-Omni-30B-A3B-Instruct

Technical Overview

Qwen3‑Omni‑30B‑A3B‑Instruct is a 30‑billion‑parameter, multilingual, omni‑modal foundation model released by Qwen. It is built on a native “any‑to‑any” pipeline that can ingest and generate text, images, audio, and video within a single end‑to‑end architecture. The model is specifically tuned for interactive, real‑time streaming, so it can answer with text or natural speech the moment it receives a multimodal prompt.

• Key capabilities – multilingual text generation, image captioning, speech‑to‑text, speech‑to‑speech translation, audio captioning, video understanding, and live turn‑taking dialogue.
• Multilingual support – 119 text languages, 19 speech‑input languages, and 10 speech‑output languages (including English, Chinese, French, German, Russian, Japanese, Korean, etc.).
• MoE‑based “Thinker‑Talker” architecture – a mixture‑of‑experts design that separates reasoning (Thinker) from response generation (Talker), dramatically reducing latency while preserving the capacity of a 30 B model.
• AuT pre‑training & multi‑codebook design – unified audio‑text (AuT) pre‑training and a multi‑codebook tokeniser that minimise token‑level latency for streaming audio/video.
• Real‑time streaming – low‑latency turn‑taking, allowing the model to start speaking or typing before the full input is consumed.
• Fine‑grained control – system‑prompt conditioning lets developers steer style, tone, or domain‑specific behavior without additional fine‑tuning.

The model is packaged as a transformers‑compatible checkpoint with safetensors weights, making it easy to load via the Hugging Face pipeline API (pipeline tag: any‑to‑any). It is the flagship member of the Qwen3‑Omni family, and its companion Qwen3‑Omni‑30B‑A3B‑Captioner provides a dedicated, low‑hallucination audio‑captioning head.

Benchmark Performance

Qwen3‑Omni‑30B‑A3B‑Instruct was evaluated on a broad suite of 36 audio and video benchmarks. It achieves state‑of‑the‑art (SOTA) results on 22 of those benchmarks and open‑source SOTA on 32, surpassing most publicly available multimodal models. In particular, its automatic speech recognition (ASR) and audio‑understanding scores are on par with Google Gemini 2.5 Pro, while its video‑question‑answering and audio‑captioning metrics rank among the top‑performing systems.

• Audio benchmarks: 22/36 SOTA, open‑source SOTA on 32/36.
• Video benchmarks: competitive VQA and video‑retrieval scores, matching proprietary models.
• Multilingual ASR: comparable word‑error‑rate (WER) to Gemini 2.5 Pro across 19 input languages.

These benchmarks matter because they test the model’s ability to understand and generate across modalities in real‑world conditions—long‑form audio, noisy environments, and high‑resolution video. The strong performance demonstrates that Qwen3‑Omni can be trusted for production‑grade speech assistants, transcription services, and multimodal content creation.

Hardware Requirements

Running a 30 B MoE model with real‑time streaming demands high‑end GPU resources. The model can be sharded across multiple GPUs, but a single‑GPU inference setup still requires a large amount of VRAM due to the expert routing tables.

• VRAM – at least 48 GB of GPU memory for a full‑model load (e.g., NVIDIA A100 40 GB + 8 GB swap, or RTX 4090 24 GB with 2‑way tensor‑parallelism and off‑loading).
• Recommended GPU – NVIDIA A100 80 GB, H100 80 GB, or AMD Instinct MI250X for optimal latency and throughput.
• CPU – 16‑core Xeon or AMD EPYC processor; sufficient RAM (≥128 GB) to hold the model’s activation cache when using CPU off‑loading.
• Storage – ~30 GB for the safetensors checkpoint plus additional space for tokenizer and config files (≈5 GB). SSD/NVMe storage is recommended for fast loading.
• Performance – on a single A100 80 GB, the model can stream audio responses with < 200 ms end‑to‑end latency for 16 kHz speech; video‑question‑answering runs at ~5 tokens/second.

Use Cases

Qwen3‑Omni‑30B‑A3B‑Instruct shines in any scenario that requires simultaneous understanding of text, audio, images, or video and the ability to respond in natural language or speech. Typical applications include:

• Multilingual voice assistants – real‑time speech‑to‑text, translation, and spoken responses in up to 10 output languages.
• Audio‑driven content creation – detailed audio captioning, music analysis, and sound‑effect description for media production.
• Video‑question‑answering & tutoring – ingest a video clip and answer questions about its content, useful for e‑learning platforms.
• Customer‑support bots – handle incoming voice calls, transcribe, translate, and reply with synthesized speech.
• Accessibility tools – generate subtitles, audio descriptions, and sign‑language prompts from multimedia content.

The model can be accessed via the Hugging Face pipeline API or integrated into custom back‑ends using the model files. Its system‑prompt interface makes it easy to adapt to domain‑specific vocabularies (e.g., medical, legal, or entertainment) without retraining.

Training Details

Qwen3‑Omni‑30B‑A3B‑Instruct was trained with a three‑stage curriculum:

• Stage 1 – Text‑first pre‑training: 1 trillion tokens from a multilingual corpus (119 languages) using a standard decoder‑only objective.
• Stage 2 – Mixed multimodal training: 500 billion multimodal tokens (image‑text pairs, audio‑text pairs, video‑text clips) with the AuT framework, enabling the model to learn joint representations across modalities.
• Stage 3 – Instruction fine‑tuning: 200 billion instruction‑following examples, including system‑prompt conditioning, to align the model for conversational and task‑oriented usage.

The training employed a mixture‑of‑experts (MoE) routing mechanism with 64 experts, each with 2 B parameters, yielding an effective 30 B capacity while keeping per‑token compute modest. Training was performed on a cluster of 256 NVIDIA H100 80 GB GPUs for roughly 2 months, consuming an estimated 1.5 exaflop‑days of compute.

Fine‑tuning is supported via the standard transformers Trainer API. Users can further adapt the model to domain‑specific data (e.g., medical transcripts) by supplying custom instruction‑following datasets while preserving the MoE routing weights.

Related Papers

While the README does not list explicit arXiv links, the Qwen3‑Omni family builds on a series of publicly released works:

• Qwen: Large Language Model for Chinese and English – the original Qwen architecture.
• Thinker‑Talker: MoE‑Based Multimodal Reasoning – introduces the MoE “Thinker‑Talker” design used in Qwen3‑Omni.
• AuT: Unified Audio‑Text Pre‑training – describes the audio‑text (AuT) pre‑training pipeline.
• Multi‑Codebook Tokenizers for Low‑Latency Streaming – details the multi‑codebook approach that reduces latency.

These papers collectively provide the theoretical foundation for the model’s multimodal capabilities, MoE scaling, and streaming efficiency.

Licensing Information

The repository lists the license as other with a license_name of Apache‑2.0. In practice this means the model weights are released under the permissive Apache‑2.0 license, which grants broad rights to use, modify, and distribute the software, even for commercial purposes, provided that:

• Attribution is given to the original authors (Qwen).
• A copy of the Apache‑2.0 license is included with any redistribution.
• Any modifications are clearly marked as such.

Because the license is “other” in the Hugging Face metadata, users should double‑check the model card for any additional usage notes. Generally, the Apache‑2.0 terms allow commercial deployment, integration into SaaS products, and fine‑tuning for proprietary datasets, as long as the attribution and notice requirements are respected.