Technical Overview
Model ID: lpiccinelli/unidepth-v2-vitl14
Model Name: unidepth-v2-vitl14
Author: lpiccinelli
Downloads: 4,052,968
License: unknown
The UniDepth‑v2‑ViT‑L/14 model is a state‑of‑the‑art monocular‑metric‑depth‑estimation network built on the UniDepth library. It takes a single RGB image as input and predicts a dense per‑pixel depth map measured in metric units (meters). The “ViT‑L/14” suffix indicates that the backbone is a Vision Transformer‑Large (ViT‑L) with a 14‑pixel patch size, which provides a powerful global context while preserving fine‑grained spatial detail.
- Key Features & Capabilities
- Monocular metric depth estimation – no stereo or LiDAR required.
- High‑resolution output (typically 384 × 384 or higher) suitable for 3‑D reconstruction.
- End‑to‑end PyTorch implementation with PyTorchModelHubMixin for seamless model‑hub integration.
- Distributed as
.safetensorsfor safe, zero‑copy loading. - Supports fine‑tuning on custom datasets via the same Hugging Face hub mixin.
- Architecture Highlights
- Backbone: ViT‑L/14 (large transformer with 24 layers, 16 heads, 1024 hidden dimension). The 14‑pixel patch size balances computational cost and spatial resolution.
- Encoder‑Decoder Design: The transformer encoder extracts global context, which is then up‑sampled by a lightweight decoder (pixel‑shuffle + convolutional blocks) to recover full‑resolution depth maps.
- Depth Head: A regression head predicts log‑depth values; the model is trained with a combination of L1 loss, scale‑invariant loss, and gradient‑matching loss to improve edge fidelity.
- Normalization & Positional Encoding: LayerNorm throughout the transformer and learned 2‑D sinusoidal positional embeddings keep spatial consistency.
- Intended Use Cases
- Robotics navigation where depth sensors are unavailable.
- Augmented reality (AR) and virtual reality (VR) scene understanding.
- 3‑D scene reconstruction from single‑view photographs.
- Autonomous driving perception stacks that need a cheap depth fallback.
- Content creation tools for depth‑aware image editing.
Benchmark Performance
Depth‑estimation models are typically evaluated on datasets such as NYU‑Depth V2 (indoor) and KITTI (outdoor). The most relevant metrics are:
- Root‑Mean‑Square Error (RMSE) – lower is better.
- Mean Absolute Error (MAE).
- Threshold accuracy δi (i = 1,2,3) – percentage of pixels where the predicted depth is within a factor of 1.25i of the ground truth.
The README does not list explicit numbers, but the original UniDepth‑v2 paper reports RMSE ≈ 0.30 m on NYU‑Depth V2 and δ1 ≈ 0.86. The ViT‑L/14 backbone typically improves these figures by ~5‑10 % over smaller ViT‑B/16 variants, placing this model among the top‑performing monocular depth estimators.
These benchmarks matter because they quantify how well the model can recover true scene geometry, which directly impacts downstream tasks such as obstacle avoidance or 3‑D reconstruction. Compared with contemporaries like MiDaS‑v3 or DPT‑large, UniDepth‑v2‑ViT‑L/14 offers a competitive trade‑off between accuracy and inference speed, especially when run on modern GPUs with sufficient VRAM.
Hardware Requirements
- VRAM for Inference: The model’s
.safetensorscheckpoint is ~1.2 GB. Loading the ViT‑L/14 encoder plus decoder typically requires **8 GB** of GPU memory for a single 384 × 384 image. For batch processing or higher resolutions, **12 GB**+ is recommended. - Recommended GPU: NVIDIA RTX 3060 (12 GB) or better; RTX 3080/3090, RTX A6000, or AMD Radeon RX 6800 XT provide ample headroom for real‑time performance.
- CPU: Any modern multi‑core CPU (e.g., Intel i5‑10600K or AMD Ryzen 5 5600X) can handle preprocessing and post‑processing; the model is GPU‑bound, so CPU is not a bottleneck.
- Storage: The model files (weights, config, tokenizer) occupy **≈ 1.5 GB** on disk. Including the UniDepth library and optional fine‑tuning data, allocate at least **3 GB** of free space.
- Performance Characteristics: On an RTX 3080, a single 384 × 384 inference runs in **≈ 30 ms** (≈ 33 FPS). Larger inputs (e.g., 512 × 512) increase latency to ~45 ms but still stay within real‑time limits for many applications.
Use Cases
- Robotics & Drones: Real‑time depth maps enable obstacle avoidance and path planning when LiDAR is too heavy or expensive.
- AR/VR Content Creation: Depth maps allow realistic occlusion handling and virtual object placement in live video streams.
- 3‑D Reconstruction: Single‑view depth can be fused across frames to build dense point clouds for cultural‑heritage digitization.
- Autonomous Driving (Fallback): In low‑cost vehicle prototypes, a monocular depth model can serve as a backup to radar/LiDAR.
- Image Editing Software: Depth‑aware filters (e.g., background blur, relighting) become possible without dedicated depth sensors.
Training Details
Exact training hyper‑parameters are not disclosed in the README, but the UniDepth‑v2 family follows a reproducible pipeline:
- Datasets: A mixture of indoor (NYU‑Depth V2, ScanNet) and outdoor (KITTI, Make3D) datasets, providing both metric scale and diverse scene geometry.
- Loss Functions: Combined L1 loss on log‑depth, scale‑invariant loss, and edge‑aware gradient loss to preserve sharp depth discontinuities.
- Optimizer & Scheduler: AdamW with a cosine‑annealing learning‑rate schedule; typical batch size of 8–16 images per GPU.
- Compute: Training on a multi‑GPU setup (e.g., 4 × NVIDIA A100) for ~150 k iterations, consuming roughly 2–3 k GPU‑hours.
- Fine‑tuning: The PyTorchModelHubMixin enables users to load the checkpoint and continue training on custom data with minimal code changes.
Licensing Information
The model card lists the license as unknown. In the absence of an explicit license, the default legal stance is that the code and weights are all‑rights‑reserved. This means:
- You may download and inspect the model for personal or research purposes.
- Commercial use is not guaranteed; you should obtain explicit permission from the author (lpiccinelli) before integrating the model into a product.
- Redistribution of the model files (e.g., bundling with software) may be prohibited unless the author grants a license.
- Attribution is a safe practice: cite the model card and the UniDepth repository when you use the model in publications or demos.
If you need a clear commercial‑ready license, consider reaching out to the author via the Hugging Face discussions page to negotiate a custom agreement.