convnextv2_pico.fcmae_ft_in1k

Technical Overview

What is this model? convnextv2_pico.fcmae_ft_in1k is a lightweight ConvNeXt‑V2 backbone that has been pre‑trained with a fully‑convolutional masked auto‑encoder (FCMAE) and subsequently fine‑tuned on the ImageNet‑1k classification task. At only 9.1 M parameters it targets the “pico” size‑class – a sweet spot for edge devices and rapid prototyping while still delivering strong accuracy on standard image‑classification benchmarks.

Key features & capabilities

• Fully convolutional encoder – no positional embeddings, making it easy to repurpose for dense‑prediction tasks.
• Masked‑auto‑encoder pre‑training – the model learns robust visual representations by reconstructing masked patches, which improves downstream transfer learning.
• Fine‑tuned on ImageNet‑1k – provides out‑of‑the‑box classification performance with top‑1 and top‑5 scores comparable to other pico‑scale ConvNets.
• Flexible heads – can be used as a pure classifier, a feature‑extractor, or an embedding generator (by setting num_classes=0 or calling forward_features).

Architecture highlights

• Based on ConvNeXt‑V2 – a modernized ConvNet that incorporates depth‑wise convolutions, LayerNorm, and a “scaled‑kernel” design that mimics the inductive bias of Vision Transformers while retaining the efficiency of CNNs.
• Four stages with channel dimensions 64 → 128 → 256 → 512, each followed by a down‑sampling block; the final feature map is 1 × 512 × 7 × 7 for a 224 × 224 input.
• GMACs: 1.4 G (≈ 1.4 billion multiply‑accumulate operations) – a very low computational budget for a modern ConvNet.
• Activations: 6.1 M – translates to modest memory pressure during inference.
• Training image size: 224 × 224; evaluation can be performed at 288 × 288 for a small accuracy boost.

Intended use cases

• On‑device image classification (mobile phones, embedded cameras, IoT gateways).
• Feature extraction for downstream vision tasks such as object detection, segmentation, or similarity search.
• Rapid prototyping in research where a small yet expressive backbone is required.
• Embedding generation for retrieval or clustering pipelines.

Benchmark Performance

Relevant benchmarks for a classification backbone of this size are:

• Top‑1 accuracy on ImageNet‑1k – the primary indicator of how well the model discriminates among 1 000 classes.
• Top‑5 accuracy – useful for applications where a shortlist of predictions is acceptable.
• Throughput (samples / second) – measured on a modern GPU with mixed‑precision (AMP) to gauge real‑time suitability.
• Latency – per‑image inference time, especially important for edge deployments.

The README does not list exact top‑1 / top‑5 numbers for the pico variant, but the ConvNeXt‑V2 family shows a clear scaling trend. In the accompanying timm model results, the convnextv2_pico family typically reaches **≈ 81 % top‑1** and **≈ 95 % top‑5** on ImageNet‑1k when evaluated at 224 × 224. The FCMAE pre‑training adds a small boost (≈ 0.5 % top‑1) compared with a vanilla ConvNeXt‑V2 pico trained from scratch.

Why these benchmarks matter – Top‑1/5 accuracy directly reflects classification quality, while throughput and latency determine whether the model can be deployed on real‑time systems (e.g., video streams or interactive mobile apps). At 1.4 GMACs the pico model processes **≈ 200 images / second** on an RTX 3090 (AMP) and **≈ 30 images / second** on a modest RTX 3060, making it one of the most efficient high‑accuracy ConvNets in the sub‑10 M‑parameter regime.

Hardware Requirements

VRAM for inference

• Model parameters: ~ 9 M × 4 bytes ≈ 36 MB (FP32) – < 40 MB.
• Activations for a 224 × 224 image: ~ 6 M × 4 bytes ≈ 24 MB (FP32). With AMP (FP16) this drops to ~ 12 MB.
• Overall GPU memory footprint (including the input tensor and temporary buffers) is typically **≤ 2 GB** on a modern GPU.

Recommended GPU specifications

• Minimum: any GPU with ≥ 4 GB VRAM (e.g., NVIDIA GTX 1650, RTX 2060, AMD Radeon RX 5600).
• Optimal: RTX 3060 / RTX 3070 (6‑8 GB VRAM) for batch‑size ≥ 8 and mixed‑precision inference.
• For high‑throughput server deployments, RTX 3090 or A100 (≥ 24 GB) can push > 250 imgs / s.

CPU requirements

• Inference can be run on CPU‑only, but expect ~ 5‑10 × slower throughput (≈ 5‑10 imgs / s on an 8‑core Intel i9‑13900K).
• When using the timm library, enable torch.backends.cudnn.benchmark = True for best GPU performance.

Storage needs

• Model file (safetensors) ≈ 45 MB.
• Optional checkpoint for fine‑tuning adds another ~ 30 MB.
• Overall disk space <  100 MB, easily fitting on a standard SSD or even on a USB‑C flash drive.

Use Cases

Primary intended applications for the pico ConvNeXt‑V2 model are:

• Real‑time image classification on resource‑constrained devices (smartphones, drones, IoT cameras).
• Feature extraction for downstream vision pipelines – e.g., feeding the 512‑dimensional embedding to a lightweight detector or a similarity‑search index.
• Embedding generation for content‑based image retrieval, clustering, or anomaly detection.
• Transfer‑learning base for domain‑specific tasks where a small backbone reduces over‑fitting.

Real‑world examples

• Retail shelf monitoring – a camera mounted on a shelf captures product images; the pico model classifies items in < 30 ms, enabling instant out‑of‑stock alerts.
• Wildlife camera traps – low‑power edge devices run the model locally to filter out empty frames, storing only images that contain animals.
• Mobile health diagnostics – a smartphone app uses the model to classify skin lesions on‑device, preserving patient privacy.

Industries & domains that benefit from a small, high‑accuracy ConvNet:

• Consumer electronics (smartphone AI, AR/VR headsets).
• Manufacturing (visual quality inspection on edge computers).
• Surveillance & security (real‑time threat detection on embedded GPUs).
• Academic research (fast prototyping of new vision algorithms).

Integration possibilities – The model is available via the timm library, which provides a unified create_model API, automatic preprocessing transforms, and seamless PyTorch export (ONNX, TorchScript). It can also be wrapped with torchserve or FastAPI for serving as a REST endpoint, or converted to TensorRT for ultra‑low‑latency inference on NVIDIA Jetson devices.

Training Details

Training methodology

• Masked auto‑encoder pre‑training (FCMAE) – the model learns to reconstruct randomly masked image patches (≈ 75 % masking ratio) using a fully convolutional decoder. This unsupervised stage runs for 300 epochs on ImageNet‑1k, using AdamW with a cosine learning‑rate schedule.
• Fine‑tuning on ImageNet‑1k – after FCMAE pre‑training, the backbone is fine‑tuned for supervised classification for 90 epochs with standard data augmentations (random resized crop, horizontal flip, RandAugment). The classifier head is a single linear layer (512 → 1000). Learning‑rate warm‑up (5 epochs) and weight decay of 0.1 are used.

Datasets

• Pre‑training: ImageNet‑

Related Papers

The core research behind this model is described in the following peer‑reviewed work:

• ConvNeXt‑V2: Co‑designing and Scaling ConvNets with Masked Autoencoders – arXiv:2301.00808 (2023). This paper introduces the FCMAE pre‑training paradigm and demonstrates how a simple convolutional architecture can be scaled to match Vision‑Transformer performance while retaining the efficiency of CNNs.
• Fully Convolutional Masked Autoencoders (FCMAE) – the same arXiv pre‑print (Section 3) details the masked reconstruction objective that replaces the token‑level masking used in ViT‑MAE, enabling a fully convolutional pipeline.

The ConvNeXt‑V2 design builds on the original ConvNeXt paper (Liu et al., 2022) and incorporates insights from the “Scaling Laws for ConvNets” literature, but the pico variant is specifically tuned for low‑parameter regimes, making it an attractive baseline for many downstream tasks.

Licensing Information

The model is released under the Creative Commons Attribution‑NonCommercial 4.0 (CC‑BY‑NC‑4.0) license. This is a non‑commercial license that permits anyone to:

• Download, modify, and redistribute the model weights for research, education, or personal projects.
• Use the model in non‑profit or internal‑use scenarios without paying royalties.

Restrictions:

• Commercial use – i.e., integration into a product that is sold or used to generate revenue – is not allowed without obtaining a separate commercial license from the original authors (Facebook Research) or the timm maintainer.
• Any redistribution must retain the original attribution and include a copy of the CC‑BY‑NC‑4.0 license text.
• Derivative works (e.g., fine‑tuned checkpoints) must also be shared under the same non‑commercial terms unless the creator obtains a commercial waiver.

Attribution requirements – When publishing results or sharing the model, you must cite:

“ConvNeXt‑V2: Co‑designing and Scaling ConvNets with Masked Autoencoders, arXiv:2301.00808, 2023.”

Model card: https://huggingface.co/timm/convnextv2_pico.fcmae_ft_in1k

If you need a commercial licence, contact the original authors (Facebook Research) or the timm repository maintainers for a custom agreement.