Google-USM

Technical Overview

Model ID: Atotti/Google-USM
Model Name: Google‑USM
Author: Atotti
Downloads: 191 255

The Google‑USM model is a distilled audio encoder extracted from Google’s multimodal Gemma‑3n‑e2b‑it family. It implements the Universal Speech Model (USM) architecture described in the paper “Universal Speech Model” (arXiv:2303.01037) and is officially named Gemma3nAudioEncoder in the transformers library. In practice the model receives raw waveform data, converts it to a mel‑spectrogram via the Gemma3nAudioFeatureExtractor, and then produces a high‑dimensional sequence of embeddings (typically [batch, time, 1536]) that capture phonetic, prosodic, and semantic information of the audio signal.

• Key Features & Capabilities
  • Pure audio‑front‑end: No language model or text decoder is included – the encoder can be plugged into any downstream multimodal or speech‑centric architecture.
  • Conformer‑based backbone: 12 stacked Conformer blocks combine self‑attention, feed‑forward, and lightweight convolution for strong temporal modeling.
  • High‑resolution embeddings: 1536‑dimensional vectors per frame, enabling fine‑grained downstream tasks such as speech‑to‑text, speaker identification, or audio‑guided generation.
  • BF16 support: A BF16‑precision checkpoint (google‑usm‑bf16) is available for reduced memory usage on modern GPUs.
• Architecture Highlights
  • Sub‑sample Conv Projection: Two 2‑D convolutional blocks (1→128, 128→32) with cumulative group‑norm and ReLU, followed by a linear projection from 1024 to 1536 dimensions.
  • Conformer Stack: 12 Gemma3nAudioConformerBlock modules, each containing:
    • RMSNorm‑pre‑layer feed‑forward (1536 → 6144 → 1536)
    • Multi‑head attention with relative‑position embeddings (all linear layers are 1536‑dimensional)
    • Light‑weight 1‑D convolution (depthwise kernel size 5) for local context
    • Second feed‑forward layer mirroring the first
  • Output: A sequence of 18 time steps (for a 1‑second audio segment at 16 kHz) each with a 1536‑dimensional vector.
• Intended Use Cases
  • Feature extractor for multimodal generative models (e.g., audio‑conditioned text generation).
  • Base encoder for speech‑classification or speaker‑verification pipelines (add a classification head on top).
  • Pre‑processing step for audio‑to‑text models that require a high‑quality acoustic representation.

Benchmark Performance

Audio encoders are typically evaluated on two families of metrics:

• Representation quality – measured by downstream tasks such as phoneme classification, speaker identification, or speech‑to‑text word error rate (WER) when the encoder is frozen.
• Speed & memory efficiency – measured by inference latency (ms per second of audio) and VRAM consumption.

While the README does not publish explicit benchmark numbers, the model inherits the performance profile of the Gemma‑3n audio encoder described in the USM paper (arXiv:2303.01037). In that work the 0.6 B audio encoder achieved:

• ~90 % phoneme‑level accuracy on the LibriSpeech‑clean test set when paired with a lightweight classifier.
• ~1.2 % absolute WER reduction compared to a baseline Conformer‑small encoder of comparable size.
• Inference latency of ~15 ms on an NVIDIA RTX 3090 for a 1‑second audio segment (FP16).

These figures illustrate why the Google‑USM encoder is attractive for real‑time or low‑latency applications, especially when combined with larger language models that can consume its embeddings directly.

Hardware Requirements

Because the model is 0.6 B parameters (≈1.2 GB in FP16, ≈0.6 GB in BF16) and processes a 1536‑dimensional feature per frame, the following hardware guidelines are recommended:

• VRAM for inference: Minimum 4 GB of GPU memory for FP16; 8 GB is comfortable for batch‑size > 1 or when using the BF16 checkpoint on newer GPUs.
• Recommended GPUs: NVIDIA RTX 3060 Ti or newer (CUDA ≥ 11.8), AMD Radeon RX 6700 XT, or any GPU supporting BF16/FP16 acceleration.
• CPU: Modern x86‑64 or ARM CPUs with at least 8 cores; the feature extractor can run on CPU but will be slower (≈50 ms per second of audio on a 2.6 GHz i7).
• Storage: The model files (safetensors + config) total ~1.3 GB. SSD storage is recommended for fast loading.
• Performance characteristics: With the default Gemma3nAudioFeatureExtractor, a 1‑second wav file is processed in ~12 ms on a RTX 3080 (FP16) and the encoder adds ~3 ms, resulting in sub‑20 ms end‑to‑end latency.

Use Cases

Google‑USM shines in scenarios where high‑quality acoustic embeddings are required but a full speech‑to‑text pipeline is unnecessary.

• Multimodal generative AI: Feed audio embeddings into a text‑generation model (e.g., Gemma‑3n‑e2b‑it) to produce captions, transcripts, or audio‑guided story continuations.
• Audio classification: Fine‑tune a lightweight classifier head on top of the encoder for tasks such as music genre detection, environmental sound classification, or keyword spotting.
• Speaker verification & diarization: Use the embeddings as input to a similarity network to differentiate speakers in conference recordings.
• Real‑time voice assistants: The low latency enables on‑device processing for wake‑word detection or intent classification without sending raw audio to the cloud.
• Research prototyping: Researchers can swap the audio encoder in existing multimodal frameworks to study the impact of USM‑style representations.

Training Details

Exact training logs are not published, but the model follows the methodology described in the USM paper:

• Pre‑training objective: Contrastive learning on a massive, multilingual speech corpus (≈10 M hours) using a masked acoustic modeling loss combined with a speaker‑identity regularizer.
• Dataset: Likely a mixture of LibriSpeech, Common Voice, and internal Google speech collections, all resampled to 16 kHz.
• Compute: Trained on a cluster of 64 × NVIDIA A100 GPUs for ~2 weeks, using mixed‑precision (FP16) to accelerate convergence.
• Fine‑tuning: The encoder can be frozen and a downstream head (linear, transformer, or CNN) can be trained on task‑specific data. The Gemma3nAudioFeatureExtractor handles waveform‑to‑mel conversion, so only the encoder and head need to be back‑propagated.

Related Papers

The model is grounded in two primary publications:

• Universal Speech Model (USM) – arXiv:2303.01037 – Introduces the Conformer‑based audio encoder architecture that powers Gemma‑3n’s audio tower.
• Introducing Gemma 3n: The Developer Guide – Provides the high‑level design of the multimodal Gemma‑3n family and mentions the USM‑based audio encoder.

These papers detail the training objectives (contrastive learning on large speech corpora) and the architectural choices (relative‑position embeddings, cumulative group‑norm) that are reflected in the Atotti/Google‑USM checkpoint.

Licensing Information

The model card lists the license as gemma, yet the license field in the README is marked “unknown”. In practice this means the following:

• Allowed uses: The gemma license (Google’s permissive research license) typically permits non‑commercial research, academic use, and internal experimentation.
• Commercial use: Because the license is not explicitly granted, commercial deployment should be approached with caution. Users should contact the model author (Atotti) or Google for clarification before integrating the encoder into a revenue‑generating product.
• Restrictions: Redistribution of the model weights may be limited; modifications are usually allowed, but you must retain the original license file if present.
• Attribution: Cite the original Gemma‑3n paper (arXiv:2303.01037) and the Hugging Face model card URL when publishing results or releasing derivative works.