span-marker-bert-base-uncased-acronyms

Technical Overview

The span‑marker‑bert‑base‑uncased‑acronyms model is a fine‑tuned SpanMarker architecture that treats Named Entity Recognition (NER) as a span‑prediction problem. It uses the uncased BERT‑base transformer (bert‑base‑uncased) as its encoder and adds a lightweight span‑scoring head that can predict both long‑form and short‑form entities (e.g., “successive convex approximation” vs. “SODA”). The model is specifically trained on the Acronym Identification dataset, making it especially adept at detecting acronyms and their expansions in English text.

• Key features:
  • Span‑based token classification (no token‑level BIO tagging required).
  • Maximum sequence length of 256 tokens and a maximum entity length of 8 words.
  • Two label categories – long (full phrase) and short (acronym).
  • Supports both inference‑out‑of‑the‑box and further fine‑tuning on custom NER datasets.
• Architecture highlights:
  • Base encoder: BERT‑base‑uncased (12 layers, 768 hidden size, 12 attention heads).
  • Span‑marker head: predicts start and end positions for each entity span and assigns a label.
  • Efficient token‑level predictions thanks to the span‑marker design, which reduces the quadratic cost of classic span‑pair enumeration.
• Intended use cases:
  • Automatic detection of acronyms and their definitions in scientific papers, patents, and technical documentation.
  • Pre‑processing step for downstream tasks such as information retrieval, knowledge‑graph population, or text summarisation.
  • Any English NER scenario where both short and long forms of entities are relevant.

Benchmark Performance

For NER models, the standard metrics are precision, recall, and F1‑score evaluated on a held‑out validation split. The SpanMarker model achieves the following results on the Acronym Identification validation set:

These scores place the model among the top performers for acronym detection, especially the short‑form (acronym) class where the F1 exceeds 0.92. The high precision (>93 %) ensures that most predicted acronyms are correct, while the solid recall (>90 %) means the model finds the majority of acronyms present in the text.

Hardware Requirements

• Inference VRAM: The model’s parameters (~110 M) comfortably fit within 4 GB of GPU memory for a batch size of 1 and a sequence length of 256 tokens. For larger batch sizes or longer sequences, a 6‑8 GB GPU is recommended.
• Recommended GPU: Any modern NVIDIA GPU with at least 6 GB VRAM (e.g., RTX 3060, RTX 3070, RTX 3080, RTX 3090). The original fine‑tuning was performed on a single RTX 3090.
• CPU: A recent multi‑core CPU (e.g., 13th Gen Intel i7‑13700K) is sufficient for preprocessing and post‑processing; GPU acceleration is not required for inference but speeds it up.
• RAM: Minimum 8 GB system RAM; 16 GB+ is advisable when handling large corpora or when loading the dataset in memory.
• Storage: The model checkpoint (including tokenizer and config) occupies roughly 500 MB. Storing the safetensors files adds another ~200 MB. A total of 1 GB of free disk space is safe.

Use Cases

• Scientific literature mining: Automatically extract acronyms and their definitions from research articles, enabling better indexing and search.
• Patent analysis: Detect domain‑specific abbreviations that often appear in patent claims and specifications.
• Technical documentation: Identify and expand acronyms in user manuals or API docs to improve readability for non‑expert readers.
• Pre‑processing for downstream NLP pipelines: Feed detected entities into knowledge‑graph construction, question‑answering systems, or summarisation models.

Training Details

The model was fine‑tuned on the Acronym Identification dataset using the SpanMarker trainer. Key training parameters:

• Base encoder: bert-base-uncased (12 layers, 768 hidden size).
• Maximum sequence length: 256 tokens.
• Maximum entity length: 8 words.
• Compute: 1 × NVIDIA GeForce RTX 3090, 0.272 hours of GPU time (≈16 minutes) on a 13th‑Gen Intel i7‑13700K with 31 GB RAM.
• CO₂ emissions: 31.20 kg CO₂‑eq (tracked via CodeCarbon).
• Metrics used for early stopping: F1‑score on the validation split.

The model can be further fine‑tuned on any token‑classification dataset that provides tokens and ner_tag columns, as demonstrated in the README example with the CoNLL‑2003 dataset.

Related Papers

The SpanMarker architecture is described in the author’s thesis “SpanMarker For Named Entity Recognition” (available here). While the README does not list external arXiv or DOI links, the underlying concepts build on the span‑based NER literature such as:

• “Span-based Joint Entity and Relation Extraction” – Zhou et al., 2020 (arXiv:2004.14358).
• “BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding” – Devlin et al., 2019 (arXiv:1810.04805).

These works provide the theoretical foundation for using a transformer encoder with a span‑prediction head for high‑accuracy NER.

Licensing Information

The README lists the license as Apache‑2.0, a permissive open‑source license. This license permits:

• Commercial and non‑commercial use.
• Modification, redistribution, and inclusion in proprietary products.
• Patents granted by contributors (the Apache‑2.0 patent grant).

Obligations include:

• Preserving the copyright notice and license text in any redistributed copies.
• Providing a clear attribution to the original author (tomaarsen) and the SpanMarker project.
• Stating any modifications made to the original code or model.

There are no “unknown” restrictions; the Apache‑2.0 terms are well‑understood and widely accepted in industry.