When DeepSeek launched V4 Pro in late April 2026, the model's self-reported benchmarks painted a rosy picture: on par with Anthropic's Opus 4.6 and OpenAI's GPT-5.4, both released roughly two months prior. But how does it hold up under independent scrutiny? The U.S. National Institute of Standards and Technology (NIST) just answered that question through its Center for AI Standards and Innovation (CAISI) — and the results tell a more nuanced story.
The 8-Month Gap
CAISI's evaluation, published May 2, 2026, tested DeepSeek V4 Pro across five domains: cybersecurity, software engineering, natural sciences, abstract reasoning, and mathematics. Their conclusion: DeepSeek V4 Pro lags behind the U.S. frontier by roughly 8 months.
That's a significant distinction. DeepSeek's own numbers positioned V4 Pro as a peer to models released in February and March 2026. CAISI's independent benchmarks — including non-public evaluations like PortBench (a CAISI-built software engineering test) and ARC-AGI-2's semi-private dataset — tell a different story. On those held-out benchmarks, the gap widens considerably.
Benchmark Breakdown
The CAISI evaluation used 9 benchmarks across 5 domains. Here's where DeepSeek V4 Pro stands against the leading U.S. models:
Cybersecurity (CTF-Archive-Diamond): GPT-5.5 leads at 71%, Opus 4.6 hits 46%, while DeepSeek V4 Pro manages 32% (imputed via IRT from a subset). This is the widest gap in the evaluation and suggests DeepSeek still has ground to cover on complex security reasoning.
Software Engineering: On SWE-Bench Verified, the field is tighter — GPT-5.5 at 81%, Opus 4.6 at 79%, DeepSeek V4 Pro at 74%. But on the non-public PortBench, which tests the ability to port CLI tools between programming languages, DeepSeek drops to 44% while GPT-5.5 scores 78% and Opus 4.6 reaches 60%.
Natural Sciences: DeepSeek V4 Pro holds its own here. FrontierScience shows a near-tie at 74%, and GPQA-Diamond puts it at 90% versus Opus 4.6's 91% and GPT-5.5's 96%.
Mathematics: This is DeepSeek's strongest domain. On OTIS-AIME-2025, DeepSeek V4 Pro scores 97% — beating Opus 4.6 (92%) and even approaching GPT-5.5's perfect 100%. PUMaC 2024 and SMT 2025 show similar competitiveness.
Abstract Reasoning (ARC-AGI-2 Semi-Private): DeepSeek V4 Pro scores 46% on this held-out dataset. GPT-5.5 reaches 79%, Opus 4.6 hits 63%. This is where the "8 months behind" assessment is most visible — the gap on uncontaminated reasoning tasks is substantial.
The Cost Efficiency Angle
There's a silver lining for DeepSeek. CAISI found that V4 Pro is more cost-efficient than the most competitive U.S. model (GPT-5.4 mini) on 5 out of 7 benchmarks. On those 7 benchmarks, DeepSeek V4 Pro ranged from 53% less expensive to 41% more expensive than GPT-5.4 mini. For organizations optimizing for cost-per-task rather than raw capability, DeepSeek's value proposition is real.
Why Non-Public Benchmarks Matter
One of the most important takeaways from this evaluation is the gap between public and non-public benchmark performance. DeepSeek V4 Pro looks stronger on benchmarks that are publicly available — and weaker on held-out evaluations that models can't train toward or overfit.
This doesn't necessarily mean DeepSeek is gaming benchmarks. But it does highlight why independent evaluation with uncontaminated datasets matters. Public benchmarks are useful for rough comparisons, but they can inflate a model's perceived capabilities, especially when those benchmarks have been widely available during training.
What This Means for Developers
DeepSeek V4 Pro remains the most capable open-weight model available, with MIT licensing and a 1M-token context window. For math-heavy workloads, it's genuinely competitive with frontier models. For general-purpose software engineering and cybersecurity tasks, there's a measurable gap.
The cost efficiency story matters too. If you're building applications where a 5-10% accuracy difference is acceptable in exchange for dramatically lower inference costs, DeepSeek V4 Pro deserves serious consideration. But if you need frontier-class performance on complex reasoning or security tasks, the leading U.S. models still hold a meaningful advantage.
Bottom Line
NIST's CAISI evaluation gives us something rare in AI: an independent, apples-to-apples comparison using held-out benchmarks. DeepSeek V4 Pro is a strong model — the best from China to date — but the independent data suggests it's closer to GPT-5 (released September 2025) than to GPT-5.4 or Opus 4.6 (released early 2026). The 8-month lag isn't a knock on DeepSeek so much as it is a reminder that self-reported benchmarks deserve scrutiny, and that the frontier moves fast.