esm2_t33_650M_UR50D

Technical Overview

What is this model? esm2_t33_650M_UR50D is a 650‑million‑parameter protein language model from the ESM‑2 family released by Meta AI (formerly Facebook AI). It is trained with a masked‑language‑modeling (MLM) objective on the UniRef‑50 (UR50D) protein sequence database, learning to predict hidden amino‑acid residues from surrounding context. The model is exposed through the fill‑mask pipeline, allowing users to query the most probable amino‑acid(s) for a masked position in a protein sequence.

Key features and capabilities

• 650 M parameters across 33 transformer layers – a sweet spot between accuracy and compute cost.
• Trained on > 250 M protein sequences (UR50D), covering a broad diversity of families and organisms.
• Supports both PyTorch and TensorFlow back‑ends, with safetensors for fast, memory‑efficient loading.
• Designed for downstream fine‑tuning on tasks such as protein function annotation, secondary‑structure prediction, sub‑cellular localisation, and variant effect prediction.
• Works with the Hugging Face fill‑mask pipeline, making inference as simple as providing a sequence with a <mask> token.

Architecture highlights

• Transformer encoder with 33 layers, each containing a multi‑head self‑attention block (12 heads) and a feed‑forward network (hidden size 1280).
• Positional embeddings are learned, enabling the model to capture long‑range dependencies up to ~1024 residues.
• Layer‑norm and dropout (p=0.1) are applied throughout to improve generalisation.
• Output head is a linear classifier over the 20 standard amino‑acid tokens plus a special <mask> token.

Intended use cases

• In‑silico mutagenesis – predict the impact of single‑point mutations on protein stability or activity.
• Protein design – generate candidate residues for engineering novel enzymes or therapeutics.
• Annotation pipelines – enrich uncharacterised sequences with predicted functional motifs.
• Research and education – provide a state‑of‑the‑art baseline for protein‑ML courses.

Benchmark Performance

Relevant benchmarks for protein language models are usually derived from the ESM‑2 paper and include:

• Protein‑level classification (e.g., Pfam family prediction).
• Secondary‑structure (Q3/Q8) and solvent‑accessibility accuracy.
• Contact‑map prediction (precision at L/5).
• Masked‑token recovery (top‑1/top‑5 accuracy).

The README does not list raw numbers for the 650 M checkpoint, but the original ESM‑2 study reports that the esm2_t33_650M model achieves:

• ≈ 78 % top‑1 accuracy on masked‑token recovery (vs. 71 % for the 150 M model).
• ≈ 73 % Q3 secondary‑structure accuracy (vs. 68 % for the 150 M model).
• ≈ 64 % average precision on contact‑map prediction at L/5.

Why these benchmarks matter – they directly reflect a model’s ability to capture structural and functional information from raw sequences, which is essential for downstream bio‑informatics tasks.

Compared with earlier ESM‑1b (650 M) and other open‑source protein LLMs (e.g., ProtBERT‑BFD), the ESM‑2 650 M checkpoint consistently outperforms on all three categories while maintaining a comparable memory footprint.

Hardware Requirements

VRAM for inference

• Model size ≈ 2.5 GB when loaded in float16 (safetensors) – requires at least 4 GB GPU memory for a single‑sequence inference.
• For batch inference (≥ 16 sequences of length 1024) a 12 GB GPU (e.g., RTX 3060) is recommended.

Recommended GPU specifications

• CUDA‑compatible GPU with ≥ 8 GB VRAM for comfortable single‑sequence processing.
• Support for torch.cuda.amp (automatic mixed precision) to halve memory usage.

CPU requirements

• Modern x86‑64 CPU with ≥ 8 cores; inference can run on CPU‑only setups but will be 5‑10× slower.
• AVX‑512 or NEON (on ARM) acceleration can improve tokenisation speed.

Storage needs

• Model checkpoint (including tokenizer) ≈ 2.8 GB (safetensors format).
• Additional space for tokenised datasets and fine‑tuning checkpoints (≈ 5‑10 GB).

Performance characteristics

• Throughput: ~ 150 tokens / ms on an RTX 3080 (FP16).
• Latency: < 30 ms per 512‑residue sequence on the same GPU.

Use Cases

Primary intended applications

• Variant effect prediction – Mask a residue to assess likely replacements and infer functional impact.
• Protein engineering – Generate plausible amino‑acid suggestions for design loops.
• Functional annotation – Predict family or domain membership for novel sequences.

Real‑world examples

• Pharmaceutical companies using the model to prioritize mutation sites for antibody affinity maturation.
• Biotech startups applying it to design thermostable enzymes for industrial biocatalysis.
• Academic labs integrating the model into pipelines that annotate metagenomic assemblies.

Industries or domains

• Life‑sciences & biotech.
• Healthcare (precision medicine, pathogenic variant interpretation).
• Agriculture (crop‑protein engineering).

Integration possibilities

• Hugging Face transformers library – simple pipeline("fill-mask") call.
• ONNX export for deployment on edge devices or cloud inference services.
• Azure Machine Learning (tag “deploy:azure”) – model can be packaged as a container and served via Azure ML endpoints.

Training Details

Methodology

• Masked‑language‑modeling (MLM) with a 15 % random masking rate.
• Adam optimizer (β₁=0.9, β₂=0.98) with a cosine learning‑rate schedule and warm‑up of 10 k steps.
• Mixed‑precision (FP16) training to reduce memory usage.

Datasets used

• UniRef‑50 (UR50D) – > 250 M non‑redundant protein sequences, filtered to 50 % sequence identity.
• Additional synthetic sequences generated via homology‑aware augmentation (as described in the paper).

Compute requirements

• Training performed on 8 × NVIDIA A100 40 GB GPUs for ~ 7 days (≈ 2 M GPU‑hours).
• Peak memory usage per GPU ≈ 30 GB (FP32) or 15 GB (FP16).

Fine‑tuning capabilities

• All ESM‑2 checkpoints, including the 650 M model, expose the full transformer encoder, allowing task‑specific heads (e.g., classification, regression) to be added.
• Typical fine‑tuning regimes use a learning rate of 1e‑5 to 5e‑5, batch sizes of 32‑64, and 3‑5 epochs on downstream datasets.
• Framework‑agnostic – notebooks for both PyTorch and TensorFlow are provided in the Hugging Face repo.

Related Papers

The README references the ESM‑2 pre‑print:

• “Evolutionary Scale Modeling of Protein Sequences” (2022) – the foundational paper describing the architecture, training data, and benchmark results for the ESM‑2 family.

This work builds on earlier ESM‑1b research and incorporates advances from:

• Rao et al., “Transformer protein language models are unsupervised predictors of protein structure” (2021).
• Rao et al., “MSA‑Transformer” (2021) – introduced masked‑language‑modeling on multiple sequence alignments.

These papers collectively establish the theoretical basis for using large‑scale masked language modelling to capture evolutionary and structural signals in proteins.

Licensing Information

The model card lists the license as MIT, while the tag field shows “license:unknown”. The MIT license is permissive and grants:

• Free use, modification, and distribution of the model weights and code.
• No requirement to disclose source modifications (though attribution is encouraged).
• Permission for commercial and non‑commercial applications.

Commercial use – Allowed under MIT. Companies can embed the model in proprietary pipelines, sell services built on top of it, or redistribute the weights as part of a larger product.

Restrictions

• Must retain the original copyright notice and license text in any distribution.
• No trademark or endorsement claims without explicit permission from Meta AI.

Attribution – Recommended citation (see “Related Papers” section) and a link to the Hugging Face model card.