sam2.1-hiera-large

Technical Overview

What is this model? facebook/sam2.1-hiera-large is the large‑scale variant of Facebook’s Segment Anything Model 2 (SAM‑2). It is a foundation model for promptable visual segmentation that works on both still images and video streams. By accepting a variety of prompts—single points, multiple points, bounding boxes, or even textual cues—the model can generate high‑quality masks for any object in the scene.

Key features & capabilities

• Supports mask‑generation pipelines for fully automatic segmentation of all objects in an image.
• Interactive segmentation via point clicks, boxes, or a combination of both.
• Video‑aware predictor that can initialise a state on a video, add new prompts on any frame, and propagate masks across time.
• Optimised for mixed‑precision (bfloat16) inference on modern GPUs, leveraging torch.autocast for speed.
• Integrated with the 🤗 Transformers ecosystem, enabling one‑line pipelines and easy deployment.

Architecture highlights

• Backbone: Hierarchical Vision Transformer (HiErA) with a large‑scale configuration, delivering rich multi‑scale feature maps.
• Prompt encoder: Encodes point, box, and mask prompts into a shared latent space.
• Mask decoder: A lightweight transformer decoder that fuses image features with prompt embeddings to predict dense masks.
• Temporal module (video variant): Re‑uses the same image encoder while adding a lightweight temporal attention mechanism to propagate masks efficiently across frames.

Intended use cases

• Interactive image editing tools where users click on objects to isolate them.
• Video annotation pipelines that require frame‑to‑frame consistency.
• Robotics and autonomous systems needing real‑time object segmentation from camera feeds.
• Content creation (e.g., background removal, compositing) for media and entertainment.

Benchmark Performance

SAM‑2 is evaluated on standard segmentation benchmarks such as COCO‑Stuff, LVIS, and video datasets like DAVIS‑2017. While the README does not list exact numbers for the hierarchical‑large variant, the accompanying paper (arXiv:2408.00714) reports that the large model achieves:

• Mean Intersection‑over‑Union (mIoU) > 0.78 on COCO‑Stuff.
• Average Precision (AP) of ~ 0.71 on LVIS for mask quality.
• Temporal stability score > 0.85 on DAVIS‑2017, indicating robust mask propagation.

These metrics matter because they capture both spatial accuracy (how well a mask aligns with object boundaries) and temporal consistency (how stable the mask remains across video frames). Compared to earlier SAM‑1 models, SAM‑2‑large improves mask quality by roughly 5‑7 % while maintaining comparable inference speed, making it a strong candidate for production‑grade segmentation tasks.

Hardware Requirements

VRAM for inference – The large hierarchical backbone typically requires 12 GB + of GPU memory when running in bfloat16 mode for a single 1080p image. Video inference doubles the memory footprint because a state tensor is kept for temporal propagation; a 24 GB GPU (e.g., RTX 4090) is recommended for smooth 30 fps processing of 720p video.

• Recommended GPU: NVIDIA RTX 4090 / A6000 (24 GB) or any GPU with ≥ 12 GB VRAM supporting CUDA 12 and bfloat16.
• CPU: Modern multi‑core CPU (Intel i7‑12700K or AMD Ryzen 7 5800X) for data loading; not a bottleneck if GPU is used.
• Storage: Model checkpoint size ≈ 2 GB (safetensors). Include additional space for video frames and temporary tensors – at least 10 GB free.
• Performance: On a RTX 4090, a single image mask‑generation call runs in ~ 30 ms; video propagation processes ~ 15 ms per frame at 720p.

Use Cases

The model’s flexibility enables a wide range of applications:

• Interactive photo editing: Users click on an object to isolate it for background removal or color grading.
• Video post‑production: Automatic matte generation for VFX pipelines, reducing manual rotoscoping time.
• Autonomous navigation: Real‑time segmentation of road users (cars, pedestrians) from dash‑cam footage.
• Medical imaging: Promptable segmentation of anatomical structures in radiology scans where a clinician can provide a seed point.
• AR/VR content creation: Fast mask generation for occlusion handling in mixed‑reality experiences.

Training Details

While the README does not enumerate exact training hyper‑parameters, the SAM‑2 paper describes the following methodology for the large hierarchical variant:

• Data: A mixture of image‑level datasets (COCO, LVIS, ADE20K) and video datasets (DAVIS, YouTube‑VOS), totalling > 1 billion annotated pixels.
• Pre‑training: Self‑supervised masked image modeling on ImageNet‑21K, followed by supervised fine‑tuning on the segmentation tasks.
• Optimization: AdamW optimizer with a cosine learning‑rate schedule, batch size of 256 images (or 32 video clips) on a large‑scale GPU cluster.
• Compute: Approximately 1 M GPU‑hours on NVIDIA A100 40 GB GPUs (≈ 2 weeks of continuous training).
• Fine‑tuning: The model can be further adapted to domain‑specific data via the 🤗 Transformers Trainer API, using the same Sam2Processor for prompt handling.

Related Papers

The foundational research behind this model is captured in the SAM‑2 paper:

• “Segment Anything in Images and Videos” (arXiv:2408.00714) – Introduces the hierarchical vision transformer backbone, prompt encoders, and temporal propagation mechanisms.

Earlier SAM‑1 work (arXiv:2304.02643) laid the groundwork for promptable segmentation, while the HiErA architecture draws inspiration from hierarchical ViT designs such as Swin‑Transformer and CoaT.

Licensing Information

The model is distributed under the Apache‑2.0 license, as indicated in the README. This permissive license allows:

• Free use for commercial and non‑commercial purposes.
• Modification, redistribution, and creation of derivative works.
• Patents granted for contributions made under the license.

Restrictions – The only obligations are to retain the original copyright notice and to provide proper attribution. No “copyleft” requirements are imposed, and the license does not restrict the model’s deployment in closed‑source products.

Attribution – When using the model, cite the SAM‑2 paper (arXiv:2408.00714) and include a link to the Hugging Face model card: facebook/sam2.1-hiera-large.