Technical Overview
What is this model? MoLFormer‑XL‑both‑10pct is a large‑scale chemical language model that learns representations directly from SMILES strings. Trained on a 10 % subset of both the ZINC15 and PubChem databases (≈ 110 M molecules), it captures the underlying chemistry of small molecules in a self‑supervised fashion.
Key features and capabilities
- Masked‑language‑modeling (MLM) pre‑training on SMILES, enabling contextual token embeddings.
- Linear‑attention Transformer architecture with rotary positional embeddings, offering a good trade‑off between speed and expressiveness.
- Designed for feature extraction – the frozen pooler output can be used for similarity search, clustering, or as input to downstream predictors.
- Fine‑tunable for classification or regression tasks such as solubility, toxicity, or quantum‑chemical property prediction.
Architecture highlights
- XL‑size Transformer (≈ 1 B parameters) with a linear‑attention mechanism that scales linearly with sequence length, crucial for SMILES up to 202 tokens.
- Rotary embeddings (RoPE) provide relative positional information without the quadratic cost of classic sinusoidal encodings.
- Pooler head that aggregates token‑level embeddings into a fixed‑size molecular vector, ready for downstream tasks.
Intended use cases
- Feature extraction for molecular similarity, visualization, and clustering.
- Fine‑tuning on MoleculeNet benchmarks (e.g., BBBP, HIV, QM9) or proprietary property datasets.
- Embedding generation for downstream machine‑learning pipelines (e.g., QSAR models, virtual screening).
Benchmark Performance
MoLFormer‑XL‑both‑10pct was evaluated on 11 MoleculeNet tasks, covering both classification (AUROC) and regression (MAE/RMSE). These benchmarks are standard for assessing how well a chemical language model captures property‑relevant information.
- Classification (AUROC): BBBP 91.5 %, HIV 81.3 %, BACE 86.6 %, SIDER 68.9 %, ClinTox 94.6 %, Tox21 84.5 %.
- Regression (MAE/RMSE): QM9 1.7754 eV, QM8 0.0108 eV, ESOL 0.3295 logS, FreeSolv 0.2221 kcal/mol, Lipophilicity 0.5472 logD.
These results demonstrate that even with only 10 % of the full data, the XL variant rivals or surpasses the full‑data MoLFormer‑XL on several tasks, highlighting the efficiency of the linear‑attention design. Compared to earlier MoLFormer‑Base models, the XL version shows consistent gains (e.g., +3 % AUROC on BBBP, –0.152 MAE on QM9), making it a strong candidate for high‑throughput virtual screening pipelines.
Hardware Requirements
VRAM for inference The model’s checkpoint (≈ 7 GB in safetensors) plus tokenizer and activation buffers typically require ~12 GB of GPU memory for batch‑size = 1. For larger batches, 16 GB or more is recommended.
Recommended GPU NVIDIA A100 (40 GB) or V100 (32 GB) provide ample headroom for both inference and fine‑tuning. The linear‑attention mechanism benefits from Tensor Cores, reducing latency on these GPUs.
CPU & storage A modern multi‑core CPU (≥ 8 cores) is sufficient for tokenization and data loading. The model files occupy ~8 GB (model weights + tokenizer). SSD storage is advised to keep I/O latency low during fine‑tuning.
Performance characteristics On an A100, a single‑molecule forward pass takes ~2 ms, enabling rapid generation of embeddings for millions of compounds. Fine‑tuning on a single V100 converges within 2–3 days for typical MoleculeNet tasks (batch = 64, 10 epochs).
Use Cases
Primary intended applications
- Generating fixed‑dimensional molecular embeddings for similarity search in drug discovery.
- Fine‑tuning on property prediction tasks (toxicity, solubility, quantum chemistry) to accelerate lead optimization.
- Feeding embeddings into downstream ML models (e.g., graph neural networks) as an additional feature channel.
Real‑world examples
- Pharmaceutical companies can embed millions of virtual compounds and quickly retrieve nearest neighbors to known actives.
- Materials scientists can predict band‑gap or stability by fine‑tuning the model on small curated datasets.
- Environmental agencies can screen large chemical inventories for toxicity using the pre‑trained embeddings as input to simple classifiers.
Industries & integration
- Drug discovery, biotech, and cheminformatics platforms.
- Chemical safety and regulatory compliance tools.
- Academic research labs focusing on computational chemistry.
Integration is straightforward via the Hugging Face transformers library, with a single line of code to load the model and tokenizer.
Training Details
Methodology Self‑supervised masked language modeling on canonical SMILES, with a 15 % token masking probability. The model uses a linear‑attention mechanism to keep memory usage linear in sequence length.
Datasets A 10 % subset of ZINC15 and a 10 % subset of PubChem (≈ 110 M molecules). All SMILES were canonicalized with RDKit, isomeric information stripped, and sequences longer than 202 tokens were discarded.
Compute Training was performed on 16 × NVIDIA V100 GPUs (32 GB each). The exact number of steps is not disclosed, but the hardware setup suggests several weeks of wall‑clock time.
Fine‑tuning The model can be fine‑tuned on any downstream task using the standard Trainer API from transformers. Because the architecture is compatible with the feature‑extraction pipeline, users can freeze the encoder and train a lightweight head for classification or regression.
Licensing Information
The repository lists the license as Apache‑2.0, even though the tag field mentions “unknown”. Under Apache‑2.0 you may:
- Use the model for commercial and non‑commercial purposes.
- Modify the source code and redistribute it, provided you retain the license notice.
- Include a copy of the license in any distribution.
Restrictions are minimal: you must not use the model to create a competing product that claims to be the original IBM MoLFormer without proper attribution. No warranty is provided, and you are responsible for compliance with any downstream data licenses (e.g., ZINC or PubChem).