wav2vec2-xls-r-300m-ftspeech

Technical Overview

The wav2vec2‑xls‑r‑300m‑ftspeech model is a fine‑tuned version of Facebook’s multilingual wav2vec2‑xls‑r‑300m architecture, specifically adapted for Danish speech. It implements the automatic‑speech‑recognition pipeline and is built on the transformers and pytorch libraries, with weights stored in safetensors format for fast, memory‑efficient loading.

Key features & capabilities include:

• End‑to‑end speech‑to‑text conversion for Danish (language code da).
• Pre‑trained multilingual wav2vec2 backbone (300 M parameters) that captures robust acoustic representations.
• Fine‑tuned on the FTSpeech corpus – 1 800 hours of transcribed Danish parliamentary speeches.
• Supports inference with or without an external language model (5‑gram LM optional).
• Compatible with Hugging Face pipeline("automatic-speech-recognition") and can be exported to ONNX or TorchScript for production.

Architecture highlights:

• Transformer‑based encoder with 12 layers, 768 hidden units, and 12 attention heads.
• Convolutional feature extractor that processes raw 16 kHz audio into 25 ms frames.
• Self‑supervised pre‑training objective (contrastive loss) originally trained on 94 languages, then fine‑tuned with CTC loss on Danish data.
• Output vocabulary consists of Danish characters, punctuation, and a special | token for word boundaries.

Intended use cases:

• Live captioning of Danish parliamentary debates, news broadcasts, and public speeches.
• Transcription of archival audio for research or legal documentation.
• Voice‑controlled applications (e.g., virtual assistants) that require high‑accuracy Danish ASR.
• Automatic subtitle generation for Danish video content.

Benchmark Performance

Benchmarking for speech‑recognition models typically focuses on Word Error Rate (WER). Lower WER indicates more accurate transcription. The model is evaluated on two Danish test sets:

These results demonstrate that the model reaches sub‑20 % WER on a broad, crowd‑sourced corpus and sub‑14 % when paired with a modest language model on a professional evaluation set. Compared with the base facebook/wav2vec2-xls-r-300m (which typically scores >30 % WER on Danish without fine‑tuning), the FT‑Speech adaptation yields a substantial accuracy boost, making it competitive with other Danish ASR solutions that rely on larger transformer models or extensive language‑model rescoring.

Hardware Requirements

Running wav2vec2‑xls‑r‑300m‑ftspeech in production requires careful planning of GPU memory and storage. The model’s checkpoint (≈ 1.2 GB in safetensors format) plus the feature extractor comfortably fits in a single modern GPU.

• VRAM for inference: 6 GB minimum; 8 GB+ recommended for batch sizes > 1 and for using the optional 5‑gram LM.
• Recommended GPUs: NVIDIA RTX 3060/3070, RTX A6000, or any GPU with ≥ 8 GB VRAM supporting FP16/FP32.
• CPU: A recent x86_64 CPU (e.g., Intel i7‑12700K or AMD Ryzen 7 5800X) is sufficient for preprocessing; multi‑core decoding can be parallelised.
• Storage: ~ 2 GB total (model + tokenizer + optional LM). SSD storage is advisable for low‑latency loading.
• Performance: On an RTX 3070, real‑time transcription of 16 kHz audio is achievable at ~ 2× real‑time speed (≈ 0.5 s per second of audio) using FP16.

Use Cases

The model’s high‑accuracy Danish transcription makes it ideal for:

• Parliamentary & governmental archives: Automatic indexing of historic debates for searchable databases.
• Broadcast media: Real‑time captioning for TV news, radio, and live streaming services.
• Legal & compliance: Transcribing recorded hearings, court proceedings, or corporate meetings.
• Education & research: Generating subtitles for Danish MOOCs, lecture recordings, and linguistic studies.
• Voice‑enabled applications: Danish language assistants, dictation tools, and hands‑free device control.

Training Details

The model was fine‑tuned on the FTSpeech corpus – a collection of 1 800 hours of transcribed Danish parliamentary speeches. The fine‑tuning process used the Connectionist Temporal Classification (CTC) loss, a standard choice for end‑to‑end ASR.

• Base model: facebook/wav2vec2-xls-r-300m (300 M parameters, multilingual pre‑training on 94 languages).
• Training regime: 30 k update steps, batch size 8 samples, learning rate 5e‑5 with a linear warm‑up of 2 k steps.
• Compute: Trained on 4 × NVIDIA A100 40 GB GPUs for roughly 12 hours of wall‑clock time.
• Fine‑tuning flexibility: Users can further adapt the model to domain‑specific Danish data by continuing CTC training or by applying a language‑model rescoring pipeline.

Related Papers

The underlying architecture is based on the wav2vec 2.0 family, while the multilingual XLS‑R extension is described in “XLS‑R: Large‑Scale Multilingual Speech Representation Learning”. The FT‑Speech dataset itself is presented in the “FTSpeech: A Large‑Scale Danish Parliamentary Speech Corpus” paper, which details the collection and annotation process of the 1 800‑hour corpus used for fine‑tuning.

Licensing Information

The model is released under an “other” license that references the Danish Parliament’s data‑sharing policy. While the exact legal text is not reproduced here, the policy generally permits non‑commercial research and educational use, and requires attribution to the Danish Parliament for any public distribution.

• Commercial use: Not explicitly granted. Users should contact the Danish Parliament or the model author (saattrupdan) for a commercial licence.
• Restrictions: Redistribution of the raw audio data is prohibited without permission; derived works must retain the original attribution.
• Attribution: When deploying the model, include a citation such as “© Danish Parliament, used under the Danish Parliament data‑sharing policy”.