PE-Core-L14-336

Technical Overview

Model ID: facebook/PE-Core-L14-336 – PE‑Core‑L14‑336 is a large‑scale vision encoder from Meta’s Perception Encoder (PE) family. It is a zero‑shot image‑classification model that produces high‑quality visual embeddings without task‑specific fine‑tuning. The model is built on a dual‑tower architecture (vision + text) that is trained with contrastive vision‑language learning, then further refined on synthetically generated video data to make the visual features more robust and transferable.

Key capabilities include:

• State‑of‑the‑art zero‑shot classification on ImageNet‑1K, ImageNet‑V2, ImageNet‑A and ObjectNet.
• Strong retrieval performance on both image‑to‑text (COCO‑T2I) and video‑to‑text (Kinetics‑400, VTT‑T2I) tasks.
• General‑purpose visual embeddings that can be re‑used for downstream tasks such as clustering, similarity search, and prompt‑guided generation.

Architecture highlights:

• Vision tower: 24 transformer blocks, 1024 hidden width, 4096‑dim MLP, 16 attention heads, and an 8‑head attention‑pooling block on top. The model processes images at a 336 px resolution.
• Text tower: 24 transformer blocks, 1024 hidden width, 4096‑dim MLP, 16 heads, 32‑token context length – the same backbone used for CLIP‑style alignment.
• Parameter count: ~0.32 B (vision) + ~0.31 B (text) ≈ 0.63 B total.
• Pooling strategy: An attention‑pooling block replaces the classic class token, delivering richer global descriptors.

Intended use cases are any scenario that benefits from high‑quality visual embeddings without the cost of task‑specific training: zero‑shot image classification, cross‑modal retrieval, visual‑language grounding, and as a feature extractor for downstream vision tasks.

Benchmark Performance

Zero‑shot image classification and retrieval benchmarks are the primary yardsticks for PE‑Core models because they directly measure the quality of the learned visual embeddings and their alignment with language. The README reports the following results for the PE‑Core‑L14‑336 checkpoint (336 px vision resolution):

• ImageNet‑1K (top‑1): 83.5 %
• ImageNet‑V2 (top‑1): 77.9 %
• ImageNet‑A (hard, top‑1): 89.0 %
• ObjectNet (top‑1): 84.7 %
• COCO‑T2I retrieval (R@1): 57.1 %
• Kinetics‑400 zero‑shot video classification (top‑1): 73.4 %
• VTT‑T2I retrieval (R@1): 50.3 %

These benchmarks matter because they test the model on both standard (ImageNet) and “hard” (ImageNet‑A, ObjectNet) distributions, as well as cross‑modal retrieval tasks that require tight vision‑language alignment. Compared with the smaller PE‑Core‑B16‑224 (78.4 % ImageNet‑1K) and the larger PE‑Core‑G14‑448 (85.4 % ImageNet‑1K), the L14‑336 checkpoint strikes a sweet spot between computational cost and performance, outperforming most contemporary CLIP‑style encoders of similar size.

Hardware Requirements

Inference with PE‑Core‑L14‑336 is memory‑intensive due to its 24‑layer vision transformer and attention‑pooling block. Typical VRAM needs are:

• GPU VRAM: 12 GB (e.g., NVIDIA RTX 3060 12 GB) for single‑image inference at 336 px. Batch sizes of 8–16 comfortably fit on 24 GB cards (RTX 3090, A6000).
• Recommended GPU: NVIDIA RTX A5000 / A6000 or AMD Radeon RX 7900 XTX for faster throughput; CUDA 12.4 and PyTorch 2.5.1 are officially supported.
• CPU: Any modern x86_64 CPU (Intel i7‑12700K or AMD Ryzen 7 5800X) for preprocessing; the model itself runs on GPU.
• Storage: The checkpoint is ~1.2 GB (model weights) plus ~200 MB of config files. A fast SSD (NVMe) is recommended to avoid I/O bottlenecks.
• Performance: On an RTX 4090, a single 336 px image is processed in ~5 ms (≈200 FPS) after the initial model load.

Use Cases

Because the model excels at zero‑shot classification and cross‑modal retrieval, it is well‑suited for:

• Rapid prototyping of image‑search engines where you need to index large image collections without labeling.
• Content moderation pipelines that must detect out‑of‑distribution or adversarial images (leveraging strong performance on ImageNet‑A and ObjectNet).
• Multimedia recommendation systems that match user‑written text queries to images or video frames.
• Feature extraction for downstream vision tasks (e.g., clustering, anomaly detection, transfer learning) where a high‑quality embedding space is critical.
• Augmented reality and robotics where on‑device inference at 336 px is feasible and you need robust visual perception without extensive fine‑tuning.

Training Details

Training follows a two‑stage pipeline:

• Stage 1 – Image contrastive pre‑training: The vision tower is trained on billions of image‑text pairs using a symmetric contrastive loss (similar to CLIP). The schedule includes aggressive data augmentations, mixed‑precision training, and a cosine learning‑rate decay.
• Stage 2 – Synthetic video alignment: A proprietary video synthesis engine generates temporally coherent video clips paired with captions. The model is fine‑tuned on these clips to learn robust spatio‑temporal features and to improve the attention‑pooling block.
• Datasets: Publicly available image‑text datasets (LAION‑5B, COCO) plus the internally generated synthetic video corpus.
• Compute: Training was performed on a cluster of NVIDIA A100 40 GB GPUs (≈256 GPU‑days) with mixed‑precision (FP16) and the xformers library for efficient attention.
• Fine‑tuning: The checkpoint can be fine‑tuned on downstream tasks using the same vision‑text interface (e.g., adding a linear classifier on top of the pooled embedding).

Related Papers

The model is described in the technical report “Perception Encoder: The best visual embeddings are not at the output of the network” (arXiv 2504.13181). This paper introduces the robust contrastive pre‑training recipe, the synthetic video data engine, and the attention‑pooling mechanism that differentiates PE from traditional CLIP‑style encoders. The work builds on earlier vision‑language research such as CLIP, ALIGN, and Flamingo, but demonstrates that the “core” visual representations (taken before the final classification head) are more transferable.

Licensing Information

The model card lists the Apache‑2.0 license (the license field in the README), which is a permissive open‑source license. Although the Hugging Face tag shows “unknown”, the authoritative source is the Apache‑2.0 declaration.

• Commercial use: Allowed. You may embed the model in commercial products, SaaS offerings, or downstream services.
• Modification: You may modify the source code or fine‑tune the weights without restriction.
• Attribution: Required. The license mandates that you retain the original copyright notice and provide a copy of the Apache‑2.0 license in your distribution.
• Patents: The Apache‑2.0 license includes an explicit patent grant, protecting users from patent litigation by contributors.
• Redistribution: You can redistribute the model (including modified versions) under the same Apache‑2.0 terms.

In practice, this means you can safely ship PE‑Core‑L14‑336 with your applications as long as you include the license text and credit Meta (Facebook) as the original author.