fashion-clip

Technical Overview

Fashion‑CLIP is a domain‑specialised variant of the CLIP family that learns a joint visual‑language embedding space for fashion products. Built on the ViT‑B/32 transformer backbone for images and a masked‑self‑attention transformer for text, the model is fine‑tuned on an 800 K product catalogue from Farfetch, pairing each product image (white‑background, no models) with a concatenated textual description (highlights + short description). The objective is a contrastive loss that maximises similarity for correct (image, text) pairs while pushing apart mismatched pairs.

Key features & capabilities

• Zero‑shot image classification for fashion concepts (e.g., “black shoe”, “red dress”).
• High‑quality product embeddings that can be used for retrieval, recommendation, or clustering.
• Supports English queries; the text encoder handles both short keywords and longer natural‑language descriptions.
• Compatible with Hugging Face Transformers, ONNX and Safetensors for flexible deployment.

Architecture highlights

• Image encoder: ViT‑B/32 (12 transformer layers, 32‑pixel patch size) pre‑trained on Laion‑2B‑5B data, then fine‑tuned on fashion.
• Text encoder: 12‑layer masked self‑attention transformer, sharing the same embedding dimension (512) as the image encoder.
• Contrastive training: cosine similarity between image and text embeddings, temperature‑scaled cross‑entropy loss.
• Zero‑shot pipeline: compute cosine similarity between a query embedding and candidate class embeddings to obtain class probabilities.

Intended use cases

• Product search & discovery on e‑commerce platforms.
• Automatic tagging and categorisation of new fashion items.
• Visual similarity retrieval for recommendation engines.
• Cross‑modal analytics (e.g., “show me all red shoes”).

Benchmark Performance

Fashion‑CLIP is evaluated on three public fashion benchmarks: FMNIST, KAGL, and DEEP. These datasets test zero‑shot classification on distinct product types, making them ideal for measuring how well a model generalises without task‑specific fine‑tuning.

The upgraded FashionCLIP 2.0 (fine‑tuned from laion/CLIP‑ViT‑B/32‑laion2B‑s34B‑b79K) outperforms the original OpenAI checkpoint by a clear margin, confirming that larger pre‑training data (≈5× OpenAI) and domain‑specific fine‑tuning yield stronger zero‑shot performance.

Hardware Requirements

VRAM for inference

• ViT‑B/32 model size ≈ 150 MB (safetensors). A single 8 GB GPU can run inference comfortably with batch size = 1.
• For higher‑throughput (batch ≥ 32) or multi‑label scoring, ≥ 12 GB VRAM is recommended.

Recommended GPU

• Desktop: NVIDIA RTX 3060 (12 GB) or higher.
• Data‑center: NVIDIA A100 (40 GB) for large‑scale batch processing.

CPU & storage

• CPU: any modern x86‑64 processor; inference latency is dominated by GPU.
• Storage: model files (~200 MB) plus optional ONNX export (~150 MB). SSD preferred for fast loading.

Performance characteristics

• Single image + short query: ~ 15 ms on RTX 3060.
• Batch of 64 images: ~ 80 ms (GPU‑bound).

Use Cases

Fashion‑CLIP shines in scenarios where rapid, zero‑shot understanding of new product images is required.

• Search‑by‑image: A shopper uploads a photo; the model retrieves visually similar items from the catalogue.
• Automated tagging: New arrivals are automatically labelled with attributes such as colour, style, or brand, reducing manual curation.
• Recommendation engines: Embeddings are stored in a vector database (e.g., Pinecone, Milvus) to serve “people who liked this also liked …” suggestions.
• Trend analysis: By clustering embeddings over time, analysts can detect emerging colour or silhouette trends.
• Cross‑modal advertising: Generate ad copy that matches a product image by finding the most similar textual description.

Training Details

Methodology

• Contrastive learning with a temperature‑scaled cross‑entropy loss.
• Image encoder: ViT‑B/32 initialized from laion/CLIP-ViT-B-32-laion2B-s34B-b79K.
• Text encoder: 12‑layer masked self‑attention transformer, tokenised with the same tokenizer as the original CLIP.

Dataset

• ~ 800 K product samples from the Farfetch catalogue (English descriptions, white‑background images).
• Each sample pairs a high‑resolution image with a concatenated string of “highlight” keywords and a short description.

Compute

• Training performed on 4 × NVIDIA A100 (40 GB) GPUs for roughly 12 hours.
• Batch size: 256 (128 image‑text pairs per GPU).
• Learning rate: 5e‑5 with cosine decay; AdamW optimiser.

Fine‑tuning capabilities

• The model can be further fine‑tuned on a custom fashion dataset using the same contrastive loss.
• Because the architecture follows the standard CLIP API, developers can swap the image encoder (e.g., ViT‑L/14) or the text encoder (e.g., RoBERTa) with minimal code changes.

Related Papers

Fashion‑CLIP builds on the CLIP framework and the authors’ own research on fashion‑specific fine‑tuning.

• FashionCLIP: Improving Zero‑Shot Transfer for Fashion Image Retrieval – the peer‑reviewed article that describes the dataset, methodology, and benchmark results.
• Original CLIP paper: Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021).
• Laion‑2B‑5B pre‑training: Open‑Source Large‑Scale Contrastive Language‑Image Pre‑Training (Schuhmann et al., 2022).

Licensing Information

The repository lists a MIT license in the README, yet the model card also tags the license as “unknown”. Under the MIT terms, users are free to use, copy, modify, merge, publish, distribute, sublicense, and/or sell the software, provided they include the original copyright notice and license text in all copies or substantial portions of the software.

• Commercial use: Allowed. Companies can embed Fashion‑CLIP in e‑commerce platforms, recommendation engines, or mobile apps.
• Restrictions: No warranty; the model is provided “as‑is”. Users must comply with any third‑party data licences (e.g., Farfetch dataset) that were used for fine‑tuning.
• Attribution: Include a citation to the original paper (see “Related Papers”) and retain the MIT copyright notice.