Technical Overview
What is VideoPrism‑B? VideoPrism‑B is Google DeepMind’s foundational video encoder released under the model identifier google/videoprism-base-f16r288. It is a frozen Vision‑Transformer (ViT‑B) backbone augmented with four temporal‑attention layers, producing dense spatiotemporal embeddings from raw video frames. The model is designed to be a “plug‑and‑play” encoder that can be paired with lightweight downstream classifiers, large language models, or retrieval pipelines without any task‑specific fine‑tuning.
Key features and capabilities
- Supports input videos of arbitrary length; the default training configuration uses 16 frames of size 288 × 288 × 3.
- Outputs embeddings of shape
(num_frames × 16 × 16, feature_channels), which can be reshaped to(num_frames, 16, 16, feature_channels)for spatial‑temporal analysis. - Compact 114 M‑parameter footprint (≈ 458 MB) that fits on most modern GPUs.
- Zero‑shot performance on 33 public video‑understanding benchmarks, beating prior foundation models on 31 of them.
- Compatible with the same text tokenizer (c4_en SentencePiece) used by the CoCa image encoder, enabling video‑text similarity via cosine distance.
Architecture highlights
- Backbone: ViT‑B (Vision Transformer‑Base) pre‑trained on the WebLI dataset using the CoCa recipe.
- Temporal modeling: Four factorized temporal‑attention layers inspired by the ViViT architecture, allowing the model to capture motion patterns without exploding compute.
- Positional encoding: Learned spatial and temporal positional embeddings; temporal embeddings can be interpolated to accommodate video clips longer than the training length.
- Unified video‑text variant: Although the base model is a pure video encoder, a sibling video‑text model (VideoPrism‑LvT‑B) shares the same video backbone and adds a text encoder for cross‑modal retrieval.
Intended use cases
- Video classification – feed embeddings into a shallow MLP for action recognition, scene categorisation, or event detection.
- Spatiotemporal localization – combine embeddings with a bounding‑box proposal network to pinpoint where and when an action occurs.
- Video‑text retrieval – use the cosine similarity between video embeddings and text embeddings (from the CoCa text encoder) for open‑set search.
- Feature extraction for downstream LLMs – embed video frames and feed the resulting vectors into a language model for multimodal reasoning.
Benchmark Performance
VideoPrism‑B was evaluated on 33 public video‑understanding benchmarks spanning four task categories: action recognition, video retrieval, spatiotemporal localization, and open‑set classification. The model achieved state‑of‑the‑art results on 31 of those benchmarks while remaining frozen (no fine‑tuning). Representative metrics include:
- Kinetics‑400 – top‑1 accuracy > 84 % (surpassing prior frozen ViT‑based baselines).
- Moments in Time – top‑1 accuracy ≈ 78 %.
- MSR‑Video‑Text Retrieval – Recall@1 ≈ 63 % for video‑to‑text queries.
- AVA (spatiotemporal action detection) – mAP ≈ 31 % without any detection head fine‑tuning.
These benchmarks are widely recognised as the gold standard for video understanding because they test both spatial perception and temporal dynamics across diverse domains (sports, daily activities, movie clips, etc.). VideoPrism‑B’s strong zero‑shot performance demonstrates that a single frozen encoder can generalise across tasks, reducing the need for task‑specific data collection and training.
When compared to contemporary foundation models such as CLIP‑Video (ViT‑L/14) or Flamingo‑V2, VideoPrism‑B consistently ranks higher on retrieval and localisation metrics while using fewer parameters and less VRAM, making it a cost‑effective choice for production pipelines.
Hardware Requirements
VRAM for inference
- Minimum: 8 GB GPU memory (e.g., NVIDIA RTX 3060) for a single 16‑frame clip at 288 × 288 resolution.
- Recommended: 12 GB + (e.g., RTX 3070, RTX A5000) to allow batch processing of multiple clips and to keep latency under 100 ms per clip.
GPU specifications
- CUDA ≥ 11.2, cuDNN ≥ 8.0 for optimal TensorCore utilisation.
- GPU with FP16 (bfloat16) support – VideoPrism‑B runs ~2× faster in mixed‑precision mode.
CPU and storage
- CPU: Modern multi‑core (≥ 8 cores) for data loading and preprocessing; the model itself is GPU‑bound.
- Storage: Model checkpoint size is ≈ 458 MB; keep an additional 1 GB for tokenizer files and temporary cache.
- Disk: SSD recommended for fast video frame extraction; HDD can be used but will increase preprocessing latency.
Performance characteristics
- Throughput: ~30 frames / second on a RTX 3070 (FP16) for a batch size of 1.
- Latency: ~0.03 s per 16‑frame clip (including preprocessing) on the same hardware.
- Scalability: Temporal positional embeddings are interpolated, so longer videos (e.g., 32 frames) incur only a modest linear increase in compute.
Use Cases
VideoPrism‑B is positioned as a general‑purpose video encoder. Below are concrete scenarios where it shines:
- Action recognition for sports analytics – extract embeddings from match footage and feed a lightweight classifier to detect goals, fouls, or player movements.
- Content‑based video search – compute embeddings for a video library, then retrieve relevant clips using natural‑language queries via the paired text encoder.
- Surveillance and security – use spatiotemporal embeddings to flag anomalous activities without training a separate detector for each camera.
- Multimodal storytelling – combine video embeddings with large language models to generate captions, summaries, or interactive narratives.
- Educational video indexing – embed lecture videos and enable fast retrieval of specific topics or visual concepts.
Industries that benefit include media & entertainment, e‑learning, smart city infrastructure, advertising, and any domain that requires scalable video understanding without massive annotation costs.
Training Details
Methodology
- Pre‑training uses a two‑stage approach: first, the ViT‑B image encoder is initialized from the CoCa checkpoint trained on the WebLI dataset.
- Next, four temporal‑attention layers are added and trained on a massive video‑caption corpus (≈ 36 M video‑caption pairs + 582 M raw video clips).
- Training follows the ViViT factorised scheme – spatial attention is applied per frame, followed by temporal attention across frames.
- Optimization: AdamW with cosine learning‑rate decay, mixed‑precision (FP16) on TPU v4 pods.
Datasets
- WebLI (image‑text) – provides the initial visual‑language grounding.
- Video‑caption corpus – harvested from public video platforms, covering a wide variety of domains (sports, news, cooking, etc.).
- Additional video‑only clips – used to strengthen temporal modeling without textual supervision.
Compute requirements
- Training performed on Google’s TPU v4 pods (8 × v4‑128) for roughly 12 days.
- Estimated FLOPs: ~ 1.2 × 10¹⁵ per epoch.
Fine‑tuning capabilities
- The released checkpoint is frozen, but users can fine‑tune the entire backbone or just the final classification head on downstream datasets.
- Because the model outputs spatiotemporal embeddings, downstream tasks can be addressed with simple linear probes, reducing the need for extensive GPU resources.
Licensing Information
The model card lists the license as “unknown”, but the accompanying README explicitly states license: apache‑2.0. Apache‑2.0 is a permissive open‑source license that permits:
- Commercial use – you may embed VideoPrism‑B in proprietary products or services.
- Modification – you can adapt the model weights, fine‑tune, or integrate the encoder into larger pipelines.
- Distribution – you may share the model or derived works, provided you retain the original copyright notice.
Key requirements under Apache‑2.0:
- Preserve the NOTICE file (if present) and the copyright header in any redistributed binaries.
- Include a copy of the Apache‑2.0 license text in your distribution.
- Provide attribution to the original authors (Google DeepMind) and link to the original repository.
Because the license is permissive, there are no explicit restrictions on commercial exploitation, but you should still verify any downstream data usage policies (e.g., the training data includes web‑scraped video‑caption pairs).