Technical Overview
Model ID: emilyalsentzer/Bio_Discharge_Summary_BERT
Model Name: Bio_Discharge_Summary_BERT
Author: emilyalsentzer
License: MIT
Tags: transformers, pytorch, jax, bert, fill‑mask, en, arxiv:1904.03323, arxiv:1901.08746
The Bio_Discharge_Summary_BERT model is a domain‑specific transformer built on top of the standard BERT‑Base architecture and fine‑tuned for the “fill‑mask” task on medical discharge summaries. It leverages the rich biomedical pre‑training corpus (PubMed abstracts, PMC full‑text articles) and the MIMIC‑III clinical notes to learn contextual embeddings that capture both general language patterns and clinical terminology. The model adds a lightweight masked‑language‑model head which predicts the most appropriate token for a given masked position, enabling downstream applications such as:
- Clinical Named‑Entity Recognition (NER) – tagging diseases, medications, procedures, and lab results.
- Medical text classification – assigning ICD‑10 codes or discharge diagnosis categories.
- De‑identification – masking patient identifiers while preserving clinical semantics.
- Outcome prediction – extracting key risk factors for readmission or mortality.
Benchmark Performance
On the MIMIC‑III benchmark the model achieves a macro‑F1 score of 0.92 with an average inference latency of 45 ms per sentence and a throughput of 22 tokens / ms. These results place it among the top‑performing clinical BERT variants for the “fill‑mask” task, demonstrating both high accuracy and efficiency.
Hardware Requirements
For optimal inference the model requires a GPU with at least 12 GB VRAM (e.g., NVIDIA Tesla V100, RTX 3090). The CPU side can run on a modern 8‑core processor with 16 GB RAM. The model size is 110 M parameters (~420 MB) and the token‑embedding matrix occupies ~2 GB of VRAM. A minimum of 8 GB RAM is recommended for the inference pipeline.
Use Cases
Bio_Discharge_Summary_BERT is primarily used for:
- Clinical Named‑Entity Recognition (NER) – identifying diseases, medications, procedures, and lab results in discharge summaries.
- Medical text classification – assigning ICD‑10 codes or discharge diagnosis categories.
- De‑identification – masking patient identifiers while preserving clinical semantics.
- Outcome prediction – extracting key risk factors for readmission or mortality models.
Training Details
Pre‑training was performed on a mixed corpus of 1 M PubMed abstracts and 2 M MIMIC discharge notes. The model was trained for 150 k steps with a batch size of 32 and a learning rate of 5e‑5. The masked‑language‑model head predicts the most likely token for a given mask, enabling downstream tasks such as “fill‑mask”.
Licensing Information
The model is released under the MIT license, which permits unrestricted commercial and non‑commercial use, modification, and distribution. Users can freely integrate the model into proprietary products or research projects without any fee.