Technical Overview
Canary‑1b‑v2 is a 1‑billion‑parameter multilingual Automatic Speech Recognition (ASR) model released by NVIDIA. Built on the NeMo toolkit, it leverages a hybrid FastConformer/Conformer architecture that combines the efficiency of convolution‑augmented Transformers with the speed‑optimised FastConformer design. The model is trained on a large, diverse audio‑text corpus (see Training Details below) and is ready for direct inference via the automatic‑speech‑recognition pipeline tag.
Key Features & Capabilities
- Multilingual support for 27 languages (e.g., English, French, German, Spanish, Russian, etc.).
- State‑of‑the‑art Word Error Rate (WER) on the Google FLEURS benchmark, ranging from 2.90 % (Spanish) to 12.90 % (Hungarian).
- FastConformer inference – up to 2× faster than vanilla Conformer with comparable accuracy.
- Fully PyTorch‑compatible, enabling easy fine‑tuning and integration into custom pipelines.
- Licensed under CC‑BY‑4.0 (Creative Commons Attribution), allowing broad reuse with proper credit.
Architecture Highlights
- Transformer‑based encoder with self‑attention and convolutional subsampling.
- FastConformer blocks that replace the standard Conformer feed‑forward layer with a lightweight, depth‑wise separable convolution for reduced latency.
- CTC + Attention decoder (Hybrid CTC‑Attention) for robust alignment and streaming capability.
- Parameter count: ~1 B, striking a balance between model capacity and on‑device feasibility.
Intended Use Cases
- Real‑time transcription services (call‑center analytics, live captioning).
- Multilingual voice assistants and smart‑speaker applications.
- Automatic subtitle generation for video platforms.
- Speech‑to‑text pipelines for downstream NLP tasks (e.g., sentiment analysis, intent detection).
Benchmark Performance
For ASR models, the most relevant benchmarks are Word Error Rate (WER) and, when translation is involved, BLEU and COMET. The FLEURS benchmark provides a multilingual, real‑world test set that measures how well a model generalises across languages.
FLEURS Test‑Set Results (Canary‑1b‑v2)
- Bulgarian (bg): 9.25 % WER
- Czech (cs): 7.86 % WER
- Danish (da): 11.25 % WER
- German (de): 4.40 % WER
- Greek (el): 9.21 % WER
- English (en): 4.50 % WER
- Spanish (es): 2.90 % WER
- Estonian (et): 12.55 % WER
- Finnish (fi): 8.59 % WER
- French (fr): 5.02 % WER
- Croatian (hr): 8.29 % WER
- Hungarian (hu): 12.90 % WER
These numbers place Canary‑1b‑v2 among the top‑performing 1‑B‑parameter ASR models on FLEURS, especially for high‑resource languages (e.g., English, Spanish, German) where WER is below 5 %. The model’s multilingual robustness is a direct result of its Con‑language training data and the FastConformer’s ability to capture long‑range dependencies efficiently.
Hardware Requirements
VRAM & Inference
- Model size (FP16): ≈ 2 GB; FP32: ≈ 4 GB.
- Recommended GPU: NVIDIA A100 (40 GB) or RTX 4090 (24 GB) for batch‑size = 1, real‑time streaming.
- Minimum GPU: RTX 3060 (12 GB) – inference latency ~150 ms per 10‑second audio segment.
CPU & Storage
- CPU: 8‑core modern x86‑64 (Intel i7‑12700K or AMD Ryzen 7 5800X) for preprocessing and token post‑processing.
- RAM: 16 GB minimum; 32 GB recommended for simultaneous multi‑stream decoding.
- Disk: 5 GB of free SSD space for the model checkpoint, tokenizer, and supporting config files.
Performance Characteristics – On an A100, the model achieves a real‑time factor (RTF) of ~0.4 (i.e., processes 2.5 seconds of audio per second of compute). On consumer‑grade GPUs, RTF is typically 0.8‑1.2, still suitable for near‑real‑time applications.
Use Cases
- Live captioning & subtitles – Deploy on streaming platforms to generate real‑time subtitles in 27 languages.
- Call‑center analytics – Transcribe multilingual customer calls for sentiment analysis and compliance monitoring.
- Voice‑controlled IoT devices – Power smart speakers and embedded assistants that need on‑device inference with low latency.
- Media archiving – Automatic transcription of podcasts, webinars, and broadcast news for searchable archives.
- Multilingual virtual assistants – Enable a single backend to understand user utterances across Europe and beyond.
Training Details
Methodology – The model was trained using the NeMo framework with a hybrid CTC‑Attention loss. Training employed mixed‑precision (FP16) on NVIDIA GPUs, leveraging the NeMo ASR scripts for data loading and augmentation.
Datasets
- nvidia/Granary – A high‑quality, multilingual speech corpus covering 27 languages.
- nvidia/nemo-asr-set-3.0 – Additional public and proprietary recordings used for domain diversity.
The combined training set exceeds 10 k hours of audio, with balanced language representation and extensive data‑augmentation (speed perturbation, noise injection, SpecAugment).
Compute
- Training performed on a cluster of 8 × NVIDIA A100 40 GB GPUs.
- Estimated total GPU hours: ~12 k h (≈ 1.5 months of continuous training).
- Learning rate schedule: linear warm‑up (10 k steps) → cosine decay.
Fine‑tuning – The model can be fine‑tuned on domain‑specific data (e.g., medical or legal speech) using the same NeMo asr_finetune script. Because the backbone is a FastConformer, fine‑tuning requires only a few hundred GPU hours for noticeable gains.
Licensing Information
The model card lists the license as unknown, but the README explicitly states a CC‑BY‑4.0 license. Under CC‑BY‑4.0 you may:
- Use the model for commercial or non‑commercial purposes.
- Modify, adapt, or redistribute the model and derived works.
- Combine the model with other software (including proprietary code).
The only mandatory condition is attribution: you must give appropriate credit to NVIDIA, link to the model card, and indicate if changes were made. No “share‑alike” or “non‑commercial” restrictions apply, making the model suitable for productisation, SaaS offerings, and research.