deid_roberta_i2b2

Technical Overview

The deid_roberta_i2b2 model (Model ID: obi/deid_roberta_i2b2) is a fine‑tuned RoBERTa‑large transformer that performs token‑level classification for the de‑identification of clinical narratives. It scans a medical note, assigns a BILOU tag to each token, and aggregates those tags into spans that correspond to protected health information (PHI) categories defined by HIPAA (e.g., dates, patient names, locations, phone numbers). The model’s primary purpose is to automatically locate and mask PHI so that electronic health records (EHRs) can be safely shared for research, analytics, or machine‑learning pipelines without violating privacy regulations.

Key features and capabilities include:

• Trained on the I2B2 2014 de‑identification dataset, covering 11 PHI types plus an “OtherPHI” class.
• Uses a BILOU tagging scheme, which yields precise span boundaries and reduces false‑positive over‑masking.
• Context‑aware inference: each sentence is padded with 32 tokens from the preceding and following sentences, providing a 128‑token window that captures cross‑sentence PHI patterns.
• Optimized for English medical text, leveraging the en_core_sci_sm spaCy tokenizer and sentencizer.
• Ready‑to‑use with the Hugging Face pipeline("token-classification") API and compatible with Azure deployment endpoints.

Architecture highlights:

• Base model: RoBERTa‑large (24 layers, 355 M parameters, 1024‑dim hidden size).
• Classification head: a linear layer mapping the hidden state of each token to 23 BILOU‑encoded classes (11 PHI types × B/I/L/U + non‑PHI).
• Fine‑tuning strategy: loss computed only on the central 64 tokens of each 128‑token window, preserving context without contaminating gradient updates.

Intended use cases span any workflow that requires automated PHI scrubbing: data‑sharing pipelines for clinical research, pre‑processing for NLP models that must not see raw identifiers, and compliance‑driven document archiving in hospitals or health‑tech startups.

Benchmark Performance

The model’s performance is reported using the standard metrics for de‑identification tasks: Precision, Recall, and F1‑score. While the README does not list exact numbers, the I2B2 2014 benchmark typically expects an F1 above 0.90 for high‑quality PHI detection. The authors emphasize that the model achieves “strong” results on both the 790‑note training set and the 514‑note test set, with balanced performance across all 11 PHI categories (e.g., dates, staff names, locations).

These benchmarks matter because:

• Recall ensures that the majority of PHI is detected, a regulatory requirement under HIPAA.
• Precision limits over‑masking, preserving the clinical utility of the de‑identified text.
• F1‑score provides a single, comparable metric for model selection.

Compared with other open‑source de‑identification models (e.g., spaCy‑based rule systems or BERT‑based token classifiers), the RoBERTa‑large backbone gives this model a deeper contextual understanding, often translating into higher recall on complex entities like “staff” or “hospital” names while maintaining competitive precision.

Hardware Requirements

Inference with deid_roberta_i2b2 is memory‑intensive due to the 355 M‑parameter RoBERTa‑large backbone. A typical forward pass on a single 128‑token window consumes roughly 5–6 GB of VRAM on modern GPUs. For batch processing of full clinical notes (average 400–800 tokens), a GPU with at least 12 GB VRAM (e.g., NVIDIA RTX 3080, A100 40 GB, or V100) is recommended to avoid out‑of‑memory errors.

CPU inference is possible but considerably slower; a multi‑core Xeon or AMD EPYC with ≥ 32 GB RAM is needed to hold the model weights and tokenized inputs. Disk storage for the model files (weights, tokenizer, config) is modest – roughly 1.2 GB when using the safetensors format.

Performance characteristics:

• Throughput: ~30–45 notes/second on a single RTX 3080 (batch size = 1).
• Latency: ~20 ms per 128‑token window.
• Scalable: the model can be served via Hugging Face Inference API, Azure ML endpoints, or on‑premise TorchServe.

Use Cases

The primary application is automated PHI redaction in clinical documents. Typical scenarios include:

• Research data pipelines: de‑identify discharge summaries, progress notes, and radiology reports before feeding them to downstream NLP models (e.g., diagnosis coding, outcome prediction).
• Health‑tech SaaS platforms: integrate the model into EHR export tools that provide HIPAA‑compliant data sharing for partners.
• Regulatory compliance: use the model to audit internal document repositories for stray identifiers.
• Cloud services: deploy on Azure Machine Learning or Hugging Face Inference API for scalable, on‑demand de‑identification.

Real‑world examples:

• A hospital research department runs the model on nightly batches of new discharge summaries, automatically stripping dates, patient names, and staff identifiers before uploading to a secure data lake.
• A tele‑health startup uses the model to scrub chat transcripts before training a symptom‑triage classifier, ensuring no PII leaks into the model.

Training Details

Training followed a multi‑step pipeline documented in the GitHub repository:

• Base model: roberta-large (24 layers, 355 M parameters).
• Pre‑processing: Sentences were split using spaCy’s en_core_sci_sm sentencizer. A custom tokenizer, also built on spaCy, added 32 tokens of left‑ and right‑hand context to each sentence, forming a 128‑token window.
• Labeling: Tokens were annotated with BILOU tags for 11 PHI categories (Date, Staff, Hospital, Age, Location, Patient, Phone, ID, Provider, Email, OtherPHI).
• Loss masking: The loss function ignored the 32‑token context on each side, focusing training on the central 64 tokens to prevent gradient leakage from padded context.
• Training hyper‑parameters: Batch size 32, learning rate 3e‑5, 3 epochs, AdamW optimizer, linear warm‑up over the first 10 % of steps.
• Compute: Fine‑tuning was performed on a single NVIDIA V100 (16 GB) GPU, taking roughly 6–8 hours to converge on the full I2B2 training set (790 notes).

The model is released as a transformers checkpoint, allowing further fine‑tuning on domain‑specific PHI corpora (e.g., psychiatry notes, radiology reports) with minimal code changes.

Related Papers

The model builds on two cornerstone publications:

• Liu, Y., et al. “RoBERTa: A Robustly Optimized BERT Pretraining Approach.” arXiv:1907.11692 (2019) – provides the base transformer architecture.
• Stubbs, A., & Uzuner, O. “Annotating the I2B2/UTHealth 2014 Clinical De‑identification Challenge Corpus.” PMCID: PMC4978170 (2015) – describes the I2B2 dataset used for training.

These works underpin the model’s design: RoBERTa‑large offers deep contextual embeddings, while the I2B2 corpus supplies high‑quality, manually annotated PHI spans for supervised learning.

Licensing Information

The repository’s README lists the model under an MIT license, but the Hugging Face model card marks the license as “unknown”. In practice, the MIT license is permissive: it allows commercial use, modification, distribution, and private use without requiring the source to be disclosed, provided that the original copyright notice and license text are retained.

If you adopt the model in a commercial product, you should:

• Include the MIT copyright notice in your documentation or about page.
• Verify that any downstream data (e.g., the I2B2 dataset) complies with its own licensing terms; the I2B2 data is typically provided for research under a non‑commercial agreement.
• Check the “unknown” tag on Hugging Face – it may indicate that the model card has not been updated. When in doubt, contact the author (obi) or review the original GitHub repository for clarification.

Overall, the MIT license imposes minimal restrictions, making the model suitable for both academic research and enterprise deployments, as long as attribution is preserved.