dpt-hybrid-midas

Technical Overview

The Intel/dpt‑hybrid‑midas model is a Vision‑Transformer‑based network for monocular depth estimation. It predicts a dense depth map from a single RGB image, enabling zero‑shot inference on scenes that the model has never seen during training. The architecture builds on the Vision Transformers for Dense Prediction paper and uses a ViT‑Hybrid backbone that combines convolutional early stages with a transformer encoder, allowing the network to capture both local texture and global context.

• Key features: 1.4 M training images, zero‑shot depth transfer, high‑resolution output (interpolated to the original image size), and a lightweight “hybrid” backbone that runs efficiently on modern GPUs.
• Architecture highlights:
  • ViT‑Hybrid‑Base‑Bit‑384 backbone (convolutional stem + transformer blocks).
  • Intermediate feature aggregation from multiple transformer layers (the “hybrid” design).
  • Depth‑prediction head consisting of a series of up‑sampling convolutions that produce a single‑channel depth map.
• Intended use cases: zero‑shot depth perception for robotics, AR/VR, 3‑D reconstruction, and any computer‑vision pipeline that needs a per‑pixel distance estimate without requiring stereo or LiDAR data.

Benchmark Performance

The model is evaluated on the MIX‑6 benchmark – a composite of six depth‑estimation datasets containing roughly 1.4 M images. The README reports a Zero‑shot Transfer score of 11.06, which corresponds to the mean absolute error (MAE) after scaling the predicted depth to the ground‑truth scale. This metric is the most relevant for monocular depth models because it measures how well a model trained on a heterogeneous mix of scenes generalizes to unseen environments.

• Zero‑shot transfer (MAE ≈ 11.06) – indicates strong generalisation across indoor and outdoor scenes.
• Compared with the original DPT‑Large model (MAE ≈ 9.5) the hybrid variant trades a small drop in absolute error for reduced compute and memory footprints.
• Other contemporary models (MiDaS‑v2.1, AdaBins) typically report MAE values in the 10‑12 range on MIX‑6, placing dpt‑hybrid‑midas comfortably within the state‑of‑the‑art band.

Hardware Requirements

Inference with dpt‑hybrid‑midas is memory‑efficient enough to run on a single consumer‑grade GPU, yet it still benefits from modern Tensor‑core hardware.

• VRAM: ~2 GB for a 384 × 384 input; up to 4 GB when processing higher‑resolution images (e.g., 768 × 768) after up‑sampling.
• Recommended GPU: NVIDIA RTX 20‑series or newer (e.g., RTX 2080, RTX 3060/3070) with at least 6 GB VRAM. The model also runs on Intel Xeon Platinum 8280 CPUs (8 cores) but with slower throughput.
• CPU: Any modern x86‑64 CPU; for CPU‑only inference a minimum of 8 GB RAM is advised.
• Storage: The model checkpoint (~1 GB) plus the transformer weights (~300 MB). Total disk space < 2 GB.
• Performance: On an RTX 2080, a single 384 × 384 image processes in ~30 ms (≈30 FPS). Larger images incur a linear increase in latency.

Use Cases

The zero‑shot nature of dpt‑hybrid‑midas makes it a versatile building block for many industries.

• Robotics & autonomous navigation: Real‑time depth maps enable obstacle avoidance for drones, warehouse robots, and self‑driving cars that lack stereo sensors.
• Augmented & virtual reality: Generate scene geometry on‑the‑fly for occlusion handling and realistic object placement.
• 3‑D reconstruction & photogrammetry: Convert single‑view photos into depth‑aware point clouds for cultural‑heritage digitisation or gaming assets.
• Healthcare imaging: Estimate depth in endoscopic or laparoscopic video streams where dedicated depth sensors are unavailable.
• Geospatial analysis: Derive terrain elevation from aerial imagery for quick surveys.

Training Details

The dpt‑hybrid‑midas model was trained on the MIX‑6 dataset, a curated collection of six public depth‑estimation datasets (including NYU‑Depth V2, KITTI, and others) totaling ~1.4 M images. Training followed the protocol described in the DPT paper:

• Backbone initialisation with ImageNet‑pretrained ViT‑Hybrid weights.
• Random square crops of 384 × 384 pixels; longer side of the original image resized to 384 px before cropping.
• Data augmentations: horizontal flips, colour jitter, and random scaling.
• Loss: Scale‑invariant logarithmic depth loss combined with a gradient‑matching term to preserve edge sharpness.
• Optimiser: AdamW with a cosine‑annealing learning‑rate schedule; training ran for ~300 k iterations on a cluster of Intel Xeon Platinum 8280 CPUs paired with NVIDIA RTX 2080 GPUs.

Fine‑tuning is straightforward: replace the final depth head with a task‑specific head (e.g., for indoor‑only depth or depth‑aware segmentation) and continue training on a smaller, domain‑specific dataset. The Hugging Face Transformers library provides DPTForDepthEstimation which exposes .train() and .save_pretrained() methods for this purpose.

Related Papers

The model’s scientific foundation is anchored in two key publications:

• Vision Transformers for Dense Prediction (Ranftl et al., 2021) – introduces the DPT architecture and demonstrates its effectiveness on depth, segmentation, and surface‑normal tasks.
• MiDaS – Multi‑Scale Depth from a Single Image – the predecessor dataset and training strategy that inspired the MIX‑6 compilation used for this model.

Licensing Information

The model is released under the Apache 2.0 license, despite the “unknown” tag in the metadata. Apache 2.0 is a permissive open‑source license that grants broad rights:

• Free use for personal, academic, and commercial projects.
• Permission to modify, distribute, and create derivative works.
• Requirement to retain the original copyright notice and a copy of the license in any redistributed version.
• No warranty; the model is provided “as is”.

Because the license is permissive, you can embed the model in commercial products (e.g., robotics platforms, AR SDKs) without paying royalties, provided you include the Apache 2.0 notice.