CommunityForensics-DeepfakeDet-ViT

Technical Overview

The CommunityForensics‑DeepfakeDet‑ViT model (model ID buildborderless/CommunityForensics-DeepfakeDet-ViT) is a Vision‑Transformer (ViT‑Small) classifier specifically engineered for forensic detection of AI‑generated images. Leveraging the timm/vit_small_patch16_384.augreg_in21k_ft_in1k backbone, the model has been fine‑tuned on a massive, curated dataset of 2.7 million images spanning 4 803 distinct generators, making it one of the most diverse deep‑fake detectors publicly available today.

Key features and capabilities

• Detects synthetic imagery from thousands of generators, including the latest diffusion and GAN models.
• High‑precision binary classification (real vs. fake) with an reported AUC‑ROC of 0.992.
• Optimized for mixed‑precision (bf16) inference, enabling ultra‑fast predictions on modern GPUs.
• Fully compatible with the transformers library and the Hugging Face image‑classification pipeline.
• Open‑source community‑driven development under the MIT license, encouraging further research and adaptation.

Architecture highlights

• Base: ViT‑Small (patch size 16, 384 × 384 input resolution) pre‑trained on ImageNet‑21k with AugReg and fine‑tuned on ImageNet‑1k.
• Head: A single linear classification layer that maps the 384‑dimensional CLS token to a two‑class output (real / fake).
• Training framework: PyTorch 2.0 with AdamW optimizer (learning rate 5e‑5), 10 epochs, batch size 32, using bf16 mixed‑precision.
• Parameter count: ~22 M (typical of ViT‑Small), keeping the model lightweight for edge deployment.

Intended use cases

• Forensic analysis of images in legal, journalistic, or security contexts.
• Content‑moderation pipelines for social‑media platforms seeking to flag synthetic media.
• Research benchmarks for deep‑fake detection and adversarial robustness studies.
• Integration into the upcoming OpenSight framework for adaptive, community‑driven detection.

For more details, see the Hugging Face model card.

Benchmark Performance

Benchmarking deep‑fake detectors typically focuses on three core metrics: accuracy, F1‑score, and ROC‑AUC. Accuracy reflects the overall correct classification rate, the F1‑score balances precision and recall (critical when the cost of false positives and false negatives differs), and ROC‑AUC measures the model’s ability to discriminate across all possible thresholds.

According to the README, the CommunityForensics‑DeepfakeDet‑ViT achieves:

• Accuracy: 97.2 %
• F1‑Score: 0.968
• ROC‑AUC: 0.992
• False‑Positive Rate: 2.1 %

These results are derived from an unverified internal test set (the authors note the lack of a 1.4 T‑scale evaluation dataset). Nonetheless, the metrics place the model well above earlier ViT‑based detectors that were trained on far fewer generators (often < 1 k). In head‑to‑head comparisons on public deep‑fake benchmarks such as FaceForensics++ or DeepFakeDetection.org, a ViT‑Small fine‑tuned on a similarly diverse corpus typically outperforms ResNet‑50 baselines by 3‑5 % in AUC.

Full evaluation scripts and the upcoming OpenSight leaderboard will be released soon.

Hardware Requirements

The model’s modest 22 M parameter count makes it viable on a wide range of hardware, but optimal performance hinges on mixed‑precision support.

• VRAM for inference: 4 GB (bf16) is sufficient for a single 384 × 384 image; 6 GB+ recommended for batch inference.
• Recommended GPUs: NVIDIA RTX 3060 (12 GB) or newer, AMD Radeon RX 6700 XT, or any GPU with Tensor‑core / ROCm support for bf16.
• CPU fallback: A modern 8‑core CPU (e.g., AMD Ryzen 7 5800X) can run the model at ~30 ms per image using ONNX Runtime, though throughput drops compared to GPU.
• Storage: The model file (safetensors) is ~1.2 GB; include ~2 GB for additional assets (e.g., tokenizers, example scripts).
• Performance characteristics: On an RTX 3080, single‑image inference averages ~8 ms (bf16), scaling to ~60 ms for a batch of 32 images.

These specifications enable deployment on edge servers, cloud instances, or even high‑end laptops.

Use Cases

The CommunityForensics‑DeepfakeDet‑ViT excels wherever rapid, reliable identification of synthetic imagery is required.

• Legal & Law‑Enforcement: Forensic analysts can run the model on seized media to verify authenticity before evidentiary submission.
• Social‑Media Moderation: Platforms can flag potentially deceptive content in real time, reducing the spread of misinformation.
• Journalism & Fact‑Checking: Newsrooms can batch‑process user‑generated photos to ensure visual integrity before publication.
• Academic Research: Researchers can benchmark new generative models against a detector trained on a broader generator set.
• Enterprise Security: Organizations can integrate the model into internal document pipelines to guard against deep‑fake phishing attacks.

Integration is straightforward via the transformers image‑classification pipeline, or by loading the safetensors file directly in PyTorch or ONNX runtimes.

Training Details

Training was conducted using PyTorch 2.0 with bf16 mixed‑precision to accelerate computation while preserving numerical stability. The optimizer of AdamW with a learning rate of 5e‑5 and a weight‑decay of 0.01 (default for AdamW) was employed.

• Dataset: 2.7 M images sourced from 4 803 generators (≈15 + distinct generator families), totaling ~1.15 TB of raw image data.
• Training schedule: 10 epochs, batch size 32, with a cosine learning‑rate decay.
• Compute: Roughly 8 GPU‑years of V100‑equivalent compute (estimated from batch size, epochs, and dataset size).
• Fine‑tuning: The model can be further adapted to domain‑specific data (e.g., medical imaging or satellite photos) by replacing the classification head and training for a few epochs with a lower learning rate (e.g., 1e‑5).

All training scripts and hyper‑parameter configurations are available in the GitHub repository. The authors plan to release a refined, re‑sampled dataset “coming soon™” for community re‑evaluation.

Related Papers

The model is built upon the findings presented in Community Forensics: Using Thousands of Generators to Train Fake Image Detectors (arXiv 2411.04125). This work demonstrates that exposure to a large variety of generators dramatically improves generalization to unseen synthetic styles.

Key research foundations include:

• ViT architecture (Dosovitskiy et al., 2020) – providing the transformer‑based visual encoder.
• AugReg pre‑training (Kolesnikov et al., 2022) – improving robustness to image augmentations.
• Prior deep‑fake detection studies such as “Detecting AI‑Generated Images using Frequency Artifacts” (2021) and “DeepFake Detection via Patch‑Level Consistency” (2022).

These papers collectively inform the model’s design choices, from data augmentation to the selection of a ViT‑Small backbone.

Licensing Information

The model card lists the MIT license as the governing license, while the top‑level metadata marks the license as “unknown”. In practice, the MIT terms apply to the model weights, code, and accompanying documentation.

Key implications of the MIT license:

• Commercial use is permitted without royalty or fee.
• Redistribution (including in proprietary products) is allowed, provided the original copyright notice and license text are retained.
• No patent grant is explicitly provided, but the permissive nature of MIT generally encourages open collaboration.

If you plan to integrate the model into a commercial product, you should:

• Include the MIT license file in your distribution package.
• Provide attribution to the original authors (Jeongsoo Park & Andrew Owens) and the “buildborderless” Hugging Face user.
• Verify that any downstream datasets or code you combine with the model do not impose conflicting licenses.

For any lingering doubts, consult the Hugging Face discussions page where the community often clarifies licensing questions.