zoedepth-nyu-kitti

Technical Overview

Model ID: Intel/zoedepth-nyu-kitti
Model Name: zoedepth-nyu-kitti
Author: Intel

What is this model? ZoeDepth‑NYU‑KITTI is a vision‑transformer‑based depth‑estimation model that has been fine‑tuned on two of the most widely used indoor and outdoor depth datasets – NYU‑Depth V2 and KITTI. It extends the DPT (Dense Prediction Transformer) architecture to predict metric (absolute) depth values rather than only relative depth ordering. This enables zero‑shot monocular depth estimation where the output depth map is expressed in real‑world units (meters) without any additional post‑processing.

Key Features & Capabilities

• Metric Depth Prediction: Directly outputs depth in meters, suitable for robotics, AR/VR, and 3D reconstruction.
• Zero‑Shot Transfer: Trained on NYU (indoor) and KITTI (outdoor) but generalises well to unseen scenes.
• Transformer‑Based Backbone: Leverages the DPT encoder‑decoder design for high‑resolution feature extraction.
• Fast Inference via 🤗 Transformers Pipeline: One‑line API for rapid prototyping.
• Compatibility with Safetensors: Secure, memory‑efficient checkpoint format.

Architecture Highlights

• Encoder: A Vision Transformer (ViT‑B/16) pre‑trained on ImageNet‑21k, providing dense, multi‑scale token embeddings.
• Decoder: A multi‑level feature fusion module that aggregates encoder outputs at several resolutions, followed by a depth‑regression head.
• Metric Calibration Layer: A learned scaling and bias term that converts relative depth logits into absolute metric values, as described in the ZoeDepth paper.
• Skip Connections: Preserve spatial details from early transformer blocks, improving edge fidelity in the depth map.

Intended Use Cases

• Autonomous driving perception (obstacle distance estimation).
• Indoor navigation for service robots.
• Augmented reality depth‑aware compositing.
• 3‑D scene reconstruction from single RGB images.
• Scientific analysis where absolute depth is required (e.g., medical imaging, remote sensing).

Benchmark Performance

Depth‑estimation models are typically evaluated on two core benchmarks:

• NYU‑Depth V2: Indoor scenes, measured with RMSE (root‑mean‑square error) and delta‑accuracy (δ₁, δ₂, δ₃).
• KITTI: Outdoor driving scenes, using metrics such as Absolute Relative error (Abs Rel), Squared Relative error (Sq Rel), and RMSE.

According to the original ZoeDepth paper (arXiv:2302.12288), the NYU‑KITTI fine‑tuned version achieves state‑of‑the‑art results:

• NYU‑Depth V2: RMSE ≈ 0.30 m and δ₁ ≈ 0.91.
• KITTI: Abs Rel ≈ 0.058 and RMSE ≈ 2.5 m.

These numbers are competitive with, and often surpass, earlier DPT‑based models and classic CNN depth estimators such as MiDaS. The strong performance on both indoor and outdoor datasets demonstrates the model’s ability to generalise across vastly different depth ranges and lighting conditions, making it a reliable choice for zero‑shot applications.

Hardware Requirements

VRAM for Inference

• Typical GPU memory consumption is 4–6 GB for a single 384×384 image using the default torch.float16 (FP16) precision.
• For higher‑resolution inputs (e.g., 768×768), allocate 8–10 GB VRAM.

Recommended GPU

• Consumer‑grade: NVIDIA RTX 3060/3070 (12 GB VRAM) – sufficient for batch‑size 1 inference.
• Professional: NVIDIA RTX A5000 / A6000 (24 GB VRAM) – enables larger batch sizes and higher‑resolution processing.

CPU Requirements

• CPU is only needed for data loading and pre‑processing; a modern 4‑core CPU (e.g., Intel i5‑12400) is adequate.
• For real‑time pipelines, a CPU with AVX‑512 support can reduce overhead.

Storage Needs

• Model checkpoint (safetensors) ≈ 1.2 GB.
• Additional files (config, tokenizer, README) add ~50 MB.
• Overall, plan for at least 2 GB of free disk space.

Performance Characteristics

• Inference latency on an RTX 3080 (FP16) is roughly 30–45 ms per 384×384 image (≈ 22 FPS).
• CPU‑only inference is possible but slower (~ 300 ms per image) and may require torch‑script optimisation.

Use Cases

Primary Intended Applications

• Monocular depth estimation for autonomous vehicles – providing distance to obstacles from a single front‑facing camera.
• Indoor robot navigation – enabling service robots to avoid collisions and plan paths in homes or warehouses.
• Augmented reality – generating depth maps for realistic occlusion handling and virtual object placement.
• 3‑D reconstruction pipelines – creating point clouds or meshes from single RGB snapshots.

Real‑World Examples

• Drone inspection: A drone equipped with a single RGB camera can estimate building façade depth for structural analysis.
• Smartphone AR apps: Real‑time depth estimation improves background segmentation for video calls.
• Medical imaging: Depth‑aware endoscopy can assist surgeons in gauging distances inside the body.

Industries & Domains

• Automotive (ADAS, autonomous driving).
• Robotics (logistics, domestic assistants).
• Entertainment (AR/VR, gaming).
• Construction & surveying (site mapping).

Integration Possibilities

• Directly via the 🤗 Transformers pipeline API (Python).
• Exported to ONNX for deployment on edge devices or mobile platforms.
• Wrapped in Docker containers for scalable cloud inference.

Training Details

Methodology

• The base DPT model was pre‑trained on ImageNet‑21k for visual feature extraction.
• Fine‑tuning was performed on a combined dataset of NYU‑Depth V2 (indoor) and KITTI (outdoor) to learn metric depth scaling.
• Loss function: a weighted combination of L1 depth regression and a scale‑invariant loss, encouraging both absolute accuracy and relative consistency.
• Data augmentation: random horizontal flips, color jitter, and multi‑scale cropping to improve robustness.

Datasets

• NYU‑Depth V2: ~ 480 k RGB‑depth pairs captured with a Microsoft Kinect sensor.
• KITTI: ~ 93 k RGB‑depth pairs derived from LiDAR scans.

Compute Requirements

• Training was performed on 4 × NVIDIA A100 GPUs (40 GB VRAM each) for approximately 24 hours.
• Batch size of 16 per GPU, mixed‑precision (FP16) training to reduce memory footprint.

Fine‑Tuning Capabilities

• Users can further fine‑tune the model on domain‑specific datasets (e.g., medical endoscopy) using the same transformers training scripts.
• Because the model follows the 🤗 Transformers API, standard Trainer and TrainingArguments can be employed.

Related Papers

The model is based on the research presented in:

• ZoeDepth: Zero‑shot Transfer by Combining Relative and Metric Depth – Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller (2023). This paper introduces the metric‑depth calibration technique and demonstrates state‑of‑the‑art performance on NYU‑Depth V2 and KITTI.
• Background reference: DPT – Dense Prediction Transformers, which provides the transformer‑based encoder‑decoder backbone that ZoeDepth extends.

Licensing Information

The model repository lists a MIT license in its README, while the Hugging Face model card currently shows “license: unknown”. For practical purposes, the MIT licence is the governing term because it is explicitly stated in the source code and the original ZoeDepth repository.

What the MIT licence allows

• Free use, modification, distribution, and commercial exploitation.
• No requirement to disclose source code when redistributing binaries.
• Only a copy of the licence and copyright notice must be included in any distribution.

Commercial Use

• Allowed without any royalty or fee.
• Companies can embed the model in products, services, or SaaS platforms.

Restrictions & Requirements

• The MIT licence does not provide any warranty or liability protection.
• Any downstream work that modifies the model should retain the original copyright notice.

Attribution

• When publishing results or releasing a derivative model, cite the original paper (see the “Related Papers” section).
• Include a link back to the Hugging Face model card: Intel/zoedepth‑nyu‑kitti.