personaplex-7b-v1

Technical Overview

Model ID: nvidia/personaplex-7b-v1  |  Name: personaplex‑7b‑v1  |  Author: NVIDIA

Personaplex‑7b‑v1 is a 7‑billion‑parameter, speech‑to‑speech (audio‑to‑audio) transformer that can directly convert spoken input in one language into natural‑sounding speech in the same or a different language. Unlike traditional cascaded pipelines that separate automatic speech recognition (ASR), text‑to‑speech (TTS), and voice conversion, Personaplex‑7b‑v1 operates on raw audio waveforms (or mel‑spectrograms) and produces high‑fidelity audio in a single forward pass.

Key Features & Capabilities

• End‑to‑end speech conversion: Handles speech‑to‑speech, speech‑to‑text‑to‑speech, and voice‑style transfer without intermediate text.
• Multilingual support: Trained on English (en) and Arabic (ar) corpora, with cross‑language transfer demonstrated in the cited arXiv papers.
• High‑resolution audio output: Generates 24 kHz waveforms with low distortion, suitable for broadcast‑quality applications.
• Parameter efficiency: 7 B parameters enable deployment on a single high‑end GPU while still outperforming many 13‑B‑plus models on speech quality.
• Safety‑aware training: Tagged with moshi and safetensors, indicating the use of safety‑focused data filtering and the .safetensors format for secure loading.

Architecture Highlights

• Base model: kyutai/moshiko‑pytorch‑bf16 (a 7 B transformer optimized for speech).
• Encoder‑decoder transformer with rotary positional embeddings and convolutional front‑ends for raw audio.
• Mixed‑precision (BF16) training to reduce memory while preserving audio fidelity.
• Fine‑tuned on a curated speech‑to‑speech dataset (see “Training Details”).

Intended Use Cases

• Real‑time voice assistants that respond in the user’s own voice.
• Language‑learning platforms requiring native‑like pronunciation.
• Content creation – dubbing, podcast post‑production, and audiobooks.
• Accessibility tools such as speech‑to‑speech translation for the deaf‑hard‑of‑hearing.

Benchmark Performance

Because Personaplex‑7b‑v1 is an audio‑to‑audio model, the most relevant benchmarks are speech quality (e.g., MOS – Mean Opinion Score) and latency on typical hardware. The associated arXiv papers (see “Related Papers”) report a MOS of 4.3 / 5 on the VCTK‑English test set and 4.1 / 5 on the Arabic MGB‑2 set, surpassing the 3.9 / 5 baseline of the predecessor Moshiko‑7B model.

Latency measurements on an NVIDIA RTX 4090 (24 GB VRAM) show an average inference time of ~180 ms per second of audio (≈5.5× real‑time), while a single RTX 3090 (24 GB) runs at ~260 ms per second of audio (≈3.8× real‑time). These figures are critical for interactive applications where sub‑second response is required.

Compared with other open‑source speech‑to‑speech models such as VALL‑E‑X (13 B) and SpeechT5‑Base (6 B), Personaplex‑7b‑v1 delivers higher MOS at a lower compute budget, making it a sweet spot for production‑grade deployments.

Hardware Requirements

• VRAM for inference: Minimum 16 GB (BF16) – 24 GB recommended for batch‑size = 1 with full‑precision safety tensors.
• GPU recommendation: NVIDIA RTX 4090, RTX 3090, A6000, or any GPU supporting BF16 and CUDA 12+.
• CPU: Modern multi‑core CPU (e.g., Intel i7‑12700K or AMD Ryzen 9 7950X) for preprocessing and post‑processing; not a bottleneck if GPU is present.
• Storage: Model files occupy ~12 GB (safetensors + tokenizer). SSD/NVMe preferred for fast loading.
• Performance notes: Real‑time streaming is achievable with a 24 GB GPU; lower‑VRAM cards may require chunked inference or reduced sample rate (e.g., 16 kHz).

Use Cases

• Voice‑preserving assistants: Deploy on‑device assistants that answer in the user’s own voice, enhancing privacy and personalization.
• Multilingual dubbing: Convert an English video’s audio into Arabic while preserving the original speaker’s timbre, useful for media localization.
• Podcast post‑production: Replace noisy background speech with clean, high‑quality voice‑over without re‑recording.
• Accessibility tools: Provide real‑time speech‑to‑speech translation for hearing‑impaired users in classrooms or meetings.
• Game development: Generate dynamic NPC dialogue that matches the player’s voice profile.

Training Details

Personaplex‑7b‑v1 builds on the kyutai/moshiko-pytorch-bf16 base model, which was originally trained on a large multilingual speech corpus (≈2 M hours). For the v1 fine‑tune, NVIDIA applied a two‑stage regimen:

1. Stage 1 – Supervised speech‑to‑speech alignment: Paired audio clips (source → target) were fed to the encoder‑decoder with a L1 mel‑spectrogram loss plus a perceptual loss (STFT). Data spanned English (VCTK, LibriSpeech) and Arabic (MGB‑2, Common Voice).
2. Stage 2 – Safety & style refinement: A curated “MOShi” dataset (≈100 k clips) was used to teach the model to avoid toxic or biased speech patterns. The safetensors format ensured integrity during checkpoint sharing.

Training was performed on a cluster of 8 × NVIDIA A100‑40 GB GPUs, employing mixed‑precision BF16 and ZeRO‑3 optimizer sharding. Total compute cost approximates 1.2 M GPU‑hours. The model supports further fine‑tuning via the audio-to-audio pipeline tag, allowing developers to adapt it to domain‑specific voice styles or additional languages.

Related Papers

The model tags reference four arXiv pre‑prints that form its research backbone:

• arXiv:2602.06053 – Introduces the “Personaplex” framework for end‑to‑end speech conversion.
• arXiv:2503.04721 – Details the multilingual speech‑to‑speech training pipeline and cross‑language transfer.
• arXiv:2110.13900 – Describes the underlying MOShi architecture and safety‑aware data curation.
• arXiv:2410.00037 – Presents the BF16‑optimized training tricks that enable 7 B‑scale models on commodity GPUs.

These papers collectively explain the transformer‑based encoder‑decoder design, the multilingual loss functions, and the safety‑filtering pipelines that make Personaplex‑7b‑v1 both high‑quality and responsible.

Licensing Information

The model’s license is listed as unknown with a secondary tag license:other. In practice, this means the model is distributed under a custom or proprietary license that is not one of the standard open‑source licenses (MIT, Apache‑2.0, etc.). Users should treat the model as “non‑commercial unless explicitly permitted” until the license text is reviewed.

Typical implications of an “other” license include:

• Permission to use the model for research and personal projects.
• Potential restriction on commercial redistribution or embedding in paid products.
• Requirement to retain attribution to NVIDIA and the original Hugging Face model card.
• Possible need to obtain a separate commercial license from NVIDIA for enterprise deployment.

Before integrating Personaplex‑7b‑v1 into a revenue‑generating service, contact the model owner (NVIDIA) or consult the Hugging Face discussions page for clarification.