Technical Overview
Model ID: shi-labs/oneformer_ade20k_swin_large
Model name: oneformer_ade20k_swin_large
Author: shi‑labs
OneFormer is a universal image‑segmentation transformer that can perform three classic segmentation tasks – semantic, instance, and panoptic – with a single set of weights. This checkpoint is the “large‑size” variant that uses a Swin‑Transformer‑Large backbone and has been trained on the scene_parse_150 (ADE20K) dataset. The model follows the architecture introduced in the paper OneFormer: One Transformer to Rule Universal Image Segmentation (Jain et al., 2022).
- Key capabilities:
- Semantic segmentation – pixel‑wise class labeling for 150 ADE20K categories.
- Instance segmentation – separate masks for each object instance.
- Panoptic segmentation – unified output that combines semantic and instance masks.
- Architecture highlights:
- Backbone: Swin‑Transformer‑Large (hierarchical vision transformer with shifted windows).
- Task token mechanism – a learned token that tells the decoder which segmentation head to activate, enabling a single model to switch dynamically between tasks.
- Unified decoder – a transformer decoder that processes image features together with the task token and produces a set of class‑agnostic mask embeddings.
- Pixel‑decoder – upsamples backbone features to a high‑resolution feature map for precise mask generation.
- Intended use cases:
- Computer‑vision pipelines that need multiple segmentation modalities without maintaining separate models.
- Rapid prototyping of scene‑understanding applications (e.g., robotics, AR/VR, autonomous driving).
- Research experiments that explore multi‑task learning or task‑conditioned inference.
Benchmark Performance
The ADE20K benchmark is the de‑facto standard for evaluating universal segmentation. In the original OneFormer paper, the Swin‑Large variant achieved:
- Semantic segmentation: 57.3 % mIoU (mean Intersection‑over‑Union across 150 classes).
- Instance segmentation: 38.4 % AP (Average Precision on the instance‑level task).
- Panoptic segmentation: 46.1 % PQ (Panoptic Quality).
These numbers are competitive with task‑specific state‑of‑the‑art models while using a single checkpoint, demonstrating the efficiency of the task‑token design. The benchmarks matter because they reflect real‑world accuracy on dense prediction tasks that directly impact downstream applications such as autonomous navigation, medical imaging, and content creation.
Hardware Requirements
- VRAM for inference: The Swin‑Large backbone plus the transformer decoder typically requires ≈12 GB of GPU memory for a 512×512 input image when using FP16 precision. Larger images (e.g., 1024×1024) may need 20 GB+.
- Recommended GPU: NVIDIA RTX 3080/3090, RTX A6000, or any GPU with ≥12 GB VRAM supporting CUDA 11+ and PyTorch 2.0.
- CPU: A modern multi‑core CPU (Intel i7‑9700K or AMD Ryzen 7 3700X or newer) is sufficient for preprocessing; the heavy lifting is done on the GPU.
- Storage: The checkpoint size is ~1.2 GB (model weights + config). Allocate at least 2 GB free disk space for the model and temporary inference buffers.
- Performance: On an RTX 3090, a single forward pass for a 512×512 image takes ~70 ms (≈14 FPS) for any of the three tasks when using FP16.
Use Cases
- Robotics & autonomous systems: Real‑time scene parsing for navigation, obstacle avoidance, and manipulation.
- Augmented reality: Pixel‑accurate object segmentation to anchor virtual content on physical surfaces.
- Content creation & editing: Automatic mask generation for background removal, object extraction, and compositing in photo‑editing software.
- Medical imaging (research): Adaptable segmentation for organ/tissue delineation when fine‑tuned on domain‑specific datasets.
- Geospatial analysis: Large‑scale aerial image segmentation for land‑cover classification and urban planning.
Training Details
The checkpoint was trained on the scene_parse_150 (ADE20K) dataset, which contains 150 semantic categories and provides both instance and panoptic annotations. Training followed the protocol described in the OneFormer paper:
- Optimizer: AdamW with a weight decay of 0.05.
- Learning‑rate schedule: Linear warm‑up for the first 1500 iterations, then cosine decay.
- Batch size: 16 images per GPU (distributed training across 8 × A100 40 GB GPUs).
- Training epochs: 180 k iterations (~48 epochs on ADE20K).
- Data augmentation: Random horizontal flip, scale jitter (0.5–2.0×), and color jitter.
- Fine‑tuning: The model can be fine‑tuned on any segmentation dataset by preserving the task‑token interface; only the decoder head may need a small learning‑rate adjustment.
Licensing Information
The repository’s license field lists MIT, which is a permissive open‑source license. The Hugging Face model card currently shows the license as “unknown”, but the underlying source code and weights are released under MIT.
- Commercial use: Allowed. MIT permits integration into proprietary products without royalty.
- Restrictions: None beyond the standard MIT requirement to retain the copyright notice and license text in distributed binaries.
- Attribution: You must include the original copyright statement and a copy of the MIT license in any redistribution or derivative work.
- Patents: MIT does not provide explicit patent grants; however, the authors have not indicated any patent encumbrances for the model.