clap-htsat-fused

Technical Overview

Model ID: laion/clap-htsat-fused
Model Name: clap-htsat-fused
Author: LAION
Pipeline Tag: audio-classification

The CLAP‑HTSAT‑Fused model is a Contrastive Language‑Audio Pre‑training (CLAP) system that learns a joint embedding space for raw audio waveforms and natural‑language descriptions. By training on more than 630 k audio‑text pairs from the LAION‑Audio‑630K dataset, it can map any audio clip to a dense vector that aligns with the vector of its textual caption. The model is built on top of the CLAP architecture and incorporates a feature‑fusion mechanism that merges multiple audio encoder streams (HTSAT‑Base and a convolutional encoder) and a keyword‑to‑caption augmentation pipeline that enriches the textual side with additional descriptive tokens.

Key Features & Capabilities

• Zero‑shot audio classification – no fine‑tuning needed; simply provide candidate labels.
• Zero‑shot audio retrieval – retrieve audio clips that match a free‑form text query.
• High‑quality audio and text embeddings via ClapModel.get_audio_features and ClapModel.get_text_features.
• Variable‑length audio handling thanks to the HTSAT encoder’s transformer‑style attention.
• Supports both CPU and GPU inference with a unified ClapProcessor for preprocessing.

Architecture Highlights

• Audio Encoder: A hybrid of HTSAT‑Base (a hierarchical Transformer for audio) and a convolutional front‑end, fused together to capture both local timbral patterns and global temporal context.
• Text Encoder: A lightweight transformer (similar to DistilBERT) that converts natural‑language captions into the same embedding space.
• Contrastive Objective: The model is trained with a symmetric InfoNCE loss that pulls matching audio‑text pairs together while pushing mismatched pairs apart.
• Feature Fusion: The outputs of the two audio streams are concatenated and projected through a learned linear layer before the contrastive loss, improving robustness to diverse acoustic environments.

Intended Use Cases

• Rapid prototyping of audio classification pipelines without collecting labeled data.
• Audio search engines that accept natural‑language queries.
• Multimodal research where audio embeddings need to be combined with text or vision embeddings.
• Embedding extraction for downstream supervised tasks (e.g., sound event detection, music genre classification).

Benchmark Performance

The CLAP‑HTSAT‑Fused model was evaluated on three standard audio tasks in the original paper:

• Text‑to‑Audio Retrieval: Achieved state‑of‑the‑art mean average precision (mAP) on the LAION‑Audio‑630K held‑out set.
• Zero‑Shot Audio Classification: Outperformed prior zero‑shot baselines on ESC‑50, UrbanSound8K, and VGGSound, reaching > 85 % top‑1 accuracy on ESC‑50.
• Supervised Audio Classification (Fine‑tuned): When fine‑tuned on the same datasets, the model matched or exceeded the performance of dedicated supervised CNNs.

These benchmarks matter because they demonstrate the model’s ability to generalize across unseen sound categories and to retrieve audio based on textual intent—key capabilities for “plug‑and‑play” audio AI solutions. Compared to earlier CLAP variants that used a single audio encoder, the HTSAT‑Fused version consistently gains 2‑4 % absolute improvement in zero‑shot classification and a noticeable lift in retrieval mAP, confirming the benefit of the fusion and keyword‑augmentation strategies.

Hardware Requirements

VRAM for Inference – The model’s checkpoint (≈ 400 MB in safetensors) plus the processor tensors typically occupy ~2 GB of GPU memory for a single 10‑second audio clip. Batch processing of longer clips or larger batches may require 4‑6 GB.

Recommended GPU – Any modern NVIDIA GPU with at least 6 GB VRAM (e.g., RTX 2060, GTX 1080 Ti) will run the model comfortably. For large‑scale batch inference, a 12 GB‑plus card (RTX 3080, A100) provides headroom.

CPU Requirements – The model can be run on CPU‑only machines, but inference speed drops to ~0.5‑1 seconds per 10‑second clip on a 12‑core Xeon. Using torch.compile or ONNX Runtime can mitigate this.

Storage – The model files (weights, tokenizer, processor config) total ~450 MB. Including the Hugging Face cache and sample audio data, allocate at least 1 GB of disk space.

Performance Characteristics – On a RTX 3080, the model processes ~30 audio clips per second (batch size = 1, 10‑second clips) with torch.float16 precision. Using torch.float32 reduces throughput by ~30 % while preserving numerical stability for research.

Use Cases

Primary Applications

• Zero‑Shot Sound Event Detection: Identify sounds such as “dog bark”, “siren”, or “vacuum cleaner” without any labeled training data.
• Audio Search Engines: Index a music or sound‑effects library and retrieve clips via natural‑language queries (“rain on a window”).
• Multimodal Content Moderation: Detect prohibited audio content (e.g., gunshots) in user‑generated videos by matching against a list of textual warnings.
• Feature Extraction for Downstream ML: Feed the 512‑dimensional audio embeddings into classifiers, clustering pipelines, or similarity‑based recommendation systems.

Real‑World Examples

• Smart home assistants that classify ambient sounds to trigger automations (e.g., “doorbell ring”).
• Media asset management platforms that let editors search for sound effects using plain English.
• Wildlife monitoring projects that automatically label animal calls from field recordings.

Integration Possibilities – The model is fully compatible with the 🤗 Transformers pipeline, making it easy to embed in Python services, Flask APIs, or even mobile‑edge inference via ONNX export.

Training Details

Methodology – The model was trained with a symmetric InfoNCE loss that maximizes similarity between matching audio and caption embeddings while minimizing it for mismatched pairs. The training loop alternates between two audio encoders (HTSAT‑Base and a convolutional encoder) and fuses their outputs before the contrastive head.

Dataset – Training used the LAION‑Audio‑630K collection, which contains 633 526 audio‑text pairs spanning diverse domains (environmental sounds, music, speech, etc.). The dataset is curated to include high‑quality captions and balanced audio length distribution.

Compute – The original authors reported training on 8 × NVIDIA A100 GPUs (40 GB each) for roughly 48 hours, using mixed‑precision (FP16) to accelerate convergence. The total training FLOPs are estimated at ~1.2 × 10¹⁶.

Fine‑Tuning – While the checkpoint excels in zero‑shot settings, it can be fine‑tuned on domain‑specific labeled audio datasets (e.g., ESC‑50, UrbanSound8K) by adding a classification head on top of the frozen audio encoder or by unfreezing the entire model for end‑to‑end training. The Hugging Face ClapModel class provides get_audio_features for feature extraction and fit utilities for downstream tasks.

Related Papers

The core research behind this model is documented in the arXiv pre‑print:

• Large‑scale Contrastive Language‑Audio Pretraining with Feature Fusion and Keyword‑to‑Caption Augmentation (2022) – DOI: 10.48550/arXIV.2211.06687

This paper introduces the LAION‑Audio‑630K dataset, the contrastive training objective, and the fusion‑augmented architecture that directly powers the clap-htsat-fused checkpoint. It also benchmarks the model against prior CLAP variants and other multimodal audio embeddings, establishing the state‑of‑the‑art zero‑shot performance reported above.

Licensing Information

The model card lists the license as Apache‑2.0, while the meta‑field “License” on Hugging Face is marked “unknown”. Apache‑2.0 is a permissive open‑source license that:

• Allows commercial use, redistribution, and modification.
• Requires preservation of the original copyright notice and a copy of the license.
• Provides an explicit patent grant to downstream users.

Because the repository’s explicit license is Apache‑2.0, you may safely integrate the model into commercial products, SaaS platforms, or research pipelines, provided you include the attribution notice (see the Hugging Face model card) and retain the license file. If you encounter any conflicting “unknown” tag, it is advisable to contact the model author (LAION) for clarification before large large‑scale redistribution.