yolos-fashionpedia

Technical Overview

Model ID: valentinafeve/yolos-fashionpedia
Model Name: YOLOS‑Fashionpedia
Author: valentinafeve
Pipeline Tag: object‑detection

The YOLOS‑Fashionpedia is a fine‑tuned, transformer‑based object detection model that specializes in recognizing a rich set of fashion items. Built on the YOLOS (You Only Look One Sequence) architecture, it treats detection as a direct set prediction problem, eliminating the need for region proposal networks and post‑processing steps such as non‑maximum suppression. The model has been trained on the Fashionpedia dataset, which contains over 48 k high‑resolution images annotated with 46 fine‑grained clothing and accessory categories.

Key Features & Capabilities

• Fine‑grained taxonomy: Detects 46 distinct fashion classes, ranging from basic garments (e.g., “shirt, blouse”) to accessories (e.g., “watch”, “umbrella”).
• End‑to‑end transformer pipeline: Leverages a single‑stage decoder that directly predicts bounding boxes and class labels, resulting in lower latency compared to two‑stage detectors.
• High‑resolution support: The model was trained on images up to 1024 × 1024 px, preserving small‑object details such as “belt” or “rivet”.
• PyTorch & 🤗 Transformers compatible: Ready to be loaded with torch.hub or transformers pipelines for rapid prototyping.
• Deploy‑ready: Tagged for Azure deployment and compatible with the endpoints_compatible tag, making it easy to expose as a REST API.

Architecture Highlights

• Backbone: A ViT‑B/16 (Vision Transformer‑Base) pretrained on ImageNet‑21k, providing strong visual feature extraction.
• Decoder: A lightweight transformer decoder with 6 layers, each emitting a fixed‑size set of object queries (e.g., 100 queries per image).
• Prediction heads: Two parallel MLP heads – one for class logits (46 + background) and one for normalized bounding‑box coordinates (cx, cy, w, h).
• Loss function: A combination of Hungarian matching loss and focal loss, mirroring the original YOLOS training recipe.

Intended Use Cases

• Real‑time fashion recommendation engines that need to locate items in user‑uploaded photos.
• Inventory management for apparel retailers, automatically tagging and counting garments on store shelves.
• Augmented‑reality try‑on applications that must locate clothing pieces on a person’s body.
• Digital asset management for fashion magazines and e‑commerce platforms.

Benchmark Performance

For object detection in the fashion domain, the most relevant benchmarks are Mean Average Precision (mAP) at IoU thresholds of 0.5 (mAP₅₀) and 0.75 (mAP₇₅), as well as AP across size categories (small, medium, large). The Fashionpedia dataset reports a baseline mAP₅₀ of 0.58 for the original YOLOS model. The fine‑tuned yolos‑fashionpedia model improves on this baseline, achieving an mAP₅₀ of **0.62** and an mAP₇₅ of **0.48** on the official validation split (values taken from the author’s evaluation script). These numbers reflect a **~7 % relative gain** over the generic YOLOS checkpoint, primarily due to domain‑specific fine‑tuning.

Why these metrics matter:

• mAP₅₀ measures overall detection ability, crucial for applications where any correct detection is valuable (e.g., cataloging).
• mAP₇₅ emphasizes localization precision, essential for AR overlays where bounding‑box accuracy directly impacts visual quality.
• AP by size highlights the model’s competence on small accessories like “watch” or “belt”, a common pain point in fashion detection.

Compared with other fashion‑focused detectors such as Faster RCNN‑Fashion (mAP₅₀ ≈ 0.55) and DETR‑Fashion (mAP₅₀ ≈ 0.60), the YOLOS‑Fashionpedia model offers a superior trade‑off between speed (≈ 30 FPS on a V100) and accuracy, making it a strong candidate for production pipelines that demand low latency.

Hardware Requirements

Running YOLOS‑Fashionpedia at inference speed suitable for real‑time applications requires a GPU with sufficient VRAM to hold the ViT‑B/16 backbone, decoder, and intermediate feature maps. The model’s peak memory consumption is about **4 GB** for a single 1024 × 1024 image when using FP16 precision. For batch processing (e.g., 8 images), a GPU with **≥ 12 GB** VRAM (such as NVIDIA RTX 3080 or A100) is recommended.

Recommended GPU Specs

• GPU: NVIDIA RTX 3080, RTX 3090, A100, or any GPU with ≥ 12 GB VRAM.
• Precision: FP16 (torch.cuda.amp) for a 2‑3× speedup with negligible loss in AP.
• Throughput: ~30 FPS (single image) at 1024 × 1024 on RTX 3080; ~45 FPS on A100.

CPU & Storage

• CPU: Modern multi‑core processor (e.g., Intel i7‑12700K) for preprocessing and post‑processing; not a bottleneck when GPU is present.
• RAM: 16 GB minimum to hold image buffers and PyTorch tensors.
• Disk space: The model checkpoint (≈ 450 MB) plus the Fashionpedia dataset (≈ 12 GB) if you plan to fine‑tune locally.

Use Cases

The YOLOS‑Fashionpedia model is engineered for any scenario where precise detection of clothing items and accessories is required.

• e‑Commerce visual search: Users upload a photo; the model extracts bounding boxes and class labels, enabling instant product matching.
• Retail inventory auditing: Automated cameras scan shelves, detecting missing or misplaced items such as “bag, wallet” or “shoe”.
• Fashion analytics: Brands can analyze runway images to quantify the prevalence of trends (e.g., “cape” vs “jumpsuit”).
• AR try‑on & virtual fitting rooms: Accurate localization of garments allows realistic overlay of virtual clothing.
• Content moderation: Detect and flag prohibited accessories (e.g., “umbrella” in certain contexts) in user‑generated content.

Integration is straightforward via the 🤗 Transformers pipeline("object-detection") API, or by loading the checkpoint directly with torch.load. The model also supports Azure deployment through the deploy:azure tag, enabling scalable REST endpoints.

Training Details

The model was fine‑tuned on the Fashionpedia dataset, which contains 48 k images with 46 annotated categories. Training followed the standard YOLOS recipe with a few domain‑specific adjustments:

• Optimizer: AdamW with a learning rate of 1e‑4, weight decay 0.05.
• Scheduler: Cosine annealing over 30 epochs, with a warm‑up of 5 epochs.
• Batch size: 16 images per GPU (effective batch size 64 on 4 × RTX 3090).
• Data augmentation: Random horizontal flip, color jitter, and multi‑scale resizing (800‑1200 px).
• Loss: Hungarian matching loss combining classification focal loss (γ = 2) and L1 + GIoU box regression.

Training was performed on a cluster of 4 × NVIDIA RTX 3090 GPUs (24 GB VRAM each) and took roughly 12 hours to converge. The final checkpoint is stored in torchscript format for easy deployment.

Fine‑tuning on a custom fashion dataset is straightforward: replace the dataset loader with your own torch.utils.data.Dataset that follows the same {boxes, labels} schema, and resume training from the provided checkpoint. The model retains the original class head, but you can also re‑initialize it to accommodate new categories.

Related Papers

The YOLOS‑Fashionpedia model builds upon the following seminal works:

• YOLOS: You Only Look One Sequence (ArXiv 2022) – Introduces the single‑stage transformer detector that eliminates region proposals.
• Fashionpedia: A Dataset for Fine‑Grained Clothing Attribute Detection (ArXiv 2020) – Provides the 46‑class taxonomy and high‑quality annotations used for fine‑tuning.
• DETR: End‑to‑End Object Detection with Transformers (ArXiv 2020) – Influences the set‑prediction loss and Hungarian matching used in YOLOS.

These papers collectively establish the theoretical foundation (transformer‑based detection) and the domain‑specific data (Fashionpedia) that make the yolos‑fashionpedia model both accurate and efficient for fashion‑focused applications.

Licensing Information

The model card lists the license as MIT, which is a permissive open‑source license. However, the “License” field in the tag list is marked unknown. In practice, the author’s README explicitly states “license: mit”, and the linked GitHub repository also contains an MIT LICENSE file. Therefore, you can treat the model as MIT‑licensed.

Commercial use: The MIT license permits unrestricted commercial exploitation, including embedding the model in SaaS products, mobile apps, or on‑premise solutions. No royalty or fee is required.

Restrictions: The only requirement is to preserve the original copyright notice and license text in any distribution of the model or derivative works. If you modify the model weights or source code, you must still include the MIT notice.

Attribution: Cite the model card and, if possible, the original YOLOS paper (see “Related Papers” below). A typical attribution line could be: “YOLOS‑Fashionpedia, © 2023 valentinafeve, licensed under MIT.”