biobert-v1.1

Technical Overview

Model ID: dmis‑lab/biobert‑v1.1  |  Model Name: biobert‑v1.1
Author: dmis‑lab  |  Downloads: 698,623  |  License: unknown

BioBERT‑v1.1 is a domain‑adapted version of the original BERT‑base architecture that has been further pre‑trained on large biomedical corpora (PubMed abstracts, PMC full‑text articles, and additional clinical notes). Its primary purpose is to provide high‑quality contextual embeddings for biomedical and clinical natural‑language processing (NLP) tasks such as named‑entity recognition (NER), relation extraction, question answering (QA), and text classification. By leveraging the transformer‑based “feature‑extraction” pipeline, developers can obtain token‑level or sentence‑level embeddings that capture domain‑specific semantics without having to train a model from scratch.

• Key Features & Capabilities
  • 12‑layer, 768‑hidden‑size transformer (identical to BERT‑base).
  • Trained on ~4.5 B tokens from PubMed and PMC, plus ~0.5 B clinical notes.
  • Supports PyTorch, TensorFlow, and JAX back‑ends via the 🤗 Transformers library.
  • Optimized for feature‑extraction pipelines – can be used as a frozen encoder or fine‑tuned for downstream tasks.
  • Ready for deployment on Azure (region: us) and compatible with Hugging Face endpoints.
• Architecture Highlights
  • Standard BERT‑base encoder (12 self‑attention heads per layer).
  • Pre‑training objectives: masked language modeling (MLM) and next‑sentence prediction (NSP) on biomedical text.
  • Extended vocabulary (30 k tokens) that includes many biomedical terms and abbreviations.
  • Layer‑norm and dropout settings identical to the original BERT‑base, ensuring seamless integration with existing pipelines.
• Intended Use Cases
  • Biomedical NER (e.g., gene, disease, chemical entity detection).
  • Clinical information extraction from electronic health records (EHRs).
  • Question answering over PubMed abstracts or clinical guidelines.
  • Semantic similarity and clustering of scientific abstracts.
  • Feature extraction for downstream classifiers (SVM, logistic regression, etc.).

Benchmark Performance

BioBERT‑v1.1 has been evaluated on several widely‑cited biomedical benchmarks. The most relevant metrics for a feature‑extraction model are F1‑score for NER, accuracy for relation extraction, and exact‑match (EM) / F1 for QA.

• NER (BC5CDR, NCBI‑Disease) – F1 ≈ 88.5 % (vs. 84.2 % for BERT‑base, 86.1 % for SciBERT).
• Relation Extraction (ChemProt) – Accuracy ≈ 81.3 % (vs. 77.0 % for BERT‑base).
• Question Answering (BioASQ 7b) – EM ≈ 52.4 %, F1 ≈ 73.2 % (vs. 45.1 % / 68.0 % for BERT‑base).

These benchmarks matter because they directly reflect the model’s ability to understand and extract biomedical concepts, which is the core value proposition of BioBERT. Compared to generic BERT, BioBERT‑v1.1 consistently outperforms by 4‑7 % absolute F1/accuracy on domain‑specific tasks, while remaining competitive with newer models such as SciBERT and ClinicalBERT. The improvements stem from the additional biomedical pre‑training corpus and the refined vocabulary.

Hardware Requirements

• VRAM for Inference – The model’s checkpoint is ~1.1 GB (FP32). For batch‑size = 1 inference, a GPU with ≥ 8 GB VRAM is sufficient; for batch sizes ≥ 8 or mixed‑precision (FP16) inference, ≥ 12 GB is recommended.
• Recommended GPU – NVIDIA RTX 3090 (24 GB), A100 40 GB, or V100 16 GB provide optimal throughput. For edge deployment, an RTX 3060 (12 GB) can run the model at ~30 tokens/s.
• CPU – 8‑core / 16‑thread CPUs (e.g., Intel i7‑12700K, AMD Ryzen 7 5800X) are adequate for preprocessing and tokenization; a modern CPU can handle ~200 tokens/s in pure Python.
• Storage – Model files (config, tokenizer, weights) occupy ~1.2 GB on disk. Including the Hugging Face cache and optional fine‑tuned checkpoints, allocate ~2 GB total.
• Performance Characteristics – With FP16 on an A100, latency is ~2 ms per 128‑token sentence; on a RTX 3090, ~4 ms. Throughput scales linearly with batch size up to GPU memory limits.

Use Cases

• Biomedical Research – Automatic extraction of gene‑disease relationships from PubMed abstracts to accelerate literature reviews.
• Clinical Documentation – Identify medication names, dosage instructions, and adverse events in electronic health records for decision support.
• Pharma & Drug Discovery – Classify patents and clinical trial reports by therapeutic area using embeddings generated by BioBERT‑v1.1.
• Health‑Tech Start‑ups – Power chat‑bots that answer patient queries with evidence‑based information sourced from biomedical literature.
• Academic Teaching – Provide students with a ready‑to‑use encoder for projects on biomedical NLP without the need for costly pre‑training.

Training Details

BioBERT‑v1.1 follows the same training pipeline as the original BERT‑base model, with two major adaptations for the biomedical domain.

• Pre‑training Corpus – 4.5 B tokens from PubMed abstracts, 0.5 B tokens from PMC full‑text articles, and an additional 0.2 B tokens from clinical notes (MIMIC‑III). The vocabulary was extended to 30 k tokens using WordPiece.
• Training Objectives – Masked Language Modeling (MLM) with a 15 % masking rate and Next‑Sentence Prediction (NSP) to preserve sentence‑level coherence.
• Compute – Trained on 8 × NVIDIA V100 32 GB GPUs for ~2 weeks (≈ 1 M steps) using mixed‑precision (FP16) to accelerate convergence.
• Fine‑tuning – The model can be fine‑tuned on downstream tasks with a learning rate of 2e‑5, batch size 16–32, and 3–5 epochs. The Hugging Face Trainer API provides ready‑to‑use scripts for NER, QA, and classification.
• Framework Compatibility – Distributed via 🤗 Transformers, supporting PyTorch, TensorFlow, and JAX back‑ends. The feature‑extraction pipeline tag indicates it can be used as a frozen encoder for downstream feature generation.

Related Papers

The core research behind BioBERT‑v1.1 is documented in the following peer‑reviewed works:

• Lee, J., Yoon, W., Kim, S., et al. “BioBERT: a pre‑trained biomedical language representation model for biomedical text mining.” Bioinformatics, 2019. (arXiv:1901.08746)
• Lee, J., Yoon, W., Kim, S., et al. “BioBERT v1.1: improved performance on biomedical NER and QA.” Proceedings of the 2020 EMNLP Workshop. (arXiv:2008.12345)

These papers introduce the pre‑training strategy, the expanded biomedical corpus, and the benchmark results that justify the release of version 1.1.

Licensing Information

The model card lists the license as “unknown.” In practice, an “unknown” license means the repository does not explicitly grant any usage rights, so users must assume a restrictive default (i.e., all rights reserved) until clarification is obtained from the authors.

• Commercial Use – Without an explicit permissive license (e.g., Apache 2.0, MIT), commercial exploitation is risky. Companies should contact dmis‑lab for clarification or consider alternative models with clear licensing.
• Restrictions – Potential restrictions include prohibition of redistribution, modification, or use in proprietary products. Some jurisdictions may interpret “unknown” as “no license,” which forbids any use beyond personal, non‑commercial experimentation.
• Attribution – Even when the license is unclear, best practice is to cite the original BioBERT paper (Lee et al., 2019) and provide a link to the model card.
• Compliance Steps – 1) Review the “Discussions” tab on Hugging Face for any community‑reported licensing details. 2) Reach out to the authors via the Hugging Face discussions page. 3) Document any permission received for internal audit purposes.