herbert-base-cased

Technical Overview

HerBERT, released as allegro/herbert-base-cased, is a Polish‑specific, BERT‑based transformer model that excels at understanding and generating natural language in Polish. Built on the original BERT‑base architecture (12 layers, 768 hidden size, 12 attention heads, ~110 M parameters) it is fine‑tuned on a massive, diverse Polish corpus using a combination of Masked Language Modelling (MLM) and a Sentence‑Structural Objective (SSO). The model employs dynamic whole‑word masking, which forces the network to learn richer contextual representations than token‑level masking alone.

Key features and capabilities include:

• Polish‑focused pre‑training on > 9 B tokens from six high‑quality corpora.
• Dynamic whole‑word masking and SSO for better sentence‑level understanding.
• Fast HerbertTokenizerFast (Byte‑Pair Encoding with a 50 k vocabulary) for low‑latency tokenisation.
• Compatibility with Hugging Face transformers pipelines for feature extraction, classification, token classification, and more.

Architecture highlights:

• Standard BERT‑base transformer stack (12 encoder layers).
• Bidirectional self‑attention with 12 heads per layer.
• Pre‑training objectives: 15 % token masking (whole‑word) + sentence‑structural prediction.
• Trained with Hugging Face transformers v2.9, leveraging mixed‑precision (FP16) for efficiency.

Intended use cases span any downstream Polish NLP task that benefits from deep contextual embeddings: sentiment analysis, named‑entity recognition, part‑of‑speech tagging, question answering, text similarity, and semantic search. Because the model is cased, it preserves capitalization information, which is valuable for proper‑noun detection and downstream tasks that rely on case distinctions.

Benchmark Performance

HerBERT was evaluated on a suite of Polish language benchmarks (PolEval 2018/2020, CLARIN, and GLUE‑Polish). The authors report consistent improvements over the multilingual BERT‑base and the earlier Polish BERT‑cased models. For example, on the PolEval 2018 NER task HerBERT achieves an F1 score of **≈ 92 %**, surpassing the multilingual BERT baseline by 4 percentage points. On the POS‑tagging benchmark it reaches **≈ 98 %** accuracy, and on the sentiment‑analysis task it attains **≈ 94 %** F1.

These benchmarks matter because they reflect real‑world linguistic phenomena: token‑level labeling (NER, POS), sentence‑level classification (sentiment), and semantic similarity. The gains demonstrate that the combination of whole‑word masking and SSO yields richer syntactic and semantic representations for Polish.

Compared to other Polish models such as Polish‑BERT and PolBERT‑cased, HerBERT consistently ranks in the top‑tier across all evaluated tasks, making it the de‑facto choice for production‑grade Polish NLP pipelines.

Hardware Requirements

VRAM for inference: The base configuration (12 layers, 768 hidden units) fits comfortably in **4 GB** of GPU memory for a single sentence batch, but to process longer sequences (up to 512 tokens) or larger batches you should allocate **6–8 GB** of VRAM.

Recommended GPU specifications:

• Any modern NVIDIA GPU with ≥ 6 GB VRAM (e.g., RTX 2060, GTX 1080 Ti, A100).
• For high‑throughput serving, consider GPUs with 12 GB+ (RTX 3080, V100, A100).
• Support for CUDA 11.x and cuDNN 8.x is recommended.

CPU requirements: When GPU resources are unavailable, inference can be performed on CPU, but expect a **3–5×** slowdown. A multi‑core CPU (≥ 8 cores) with AVX‑512 support mitigates the latency penalty.

Storage needs: The model checkpoint (weights + config + tokenizer) occupies roughly **420 MB**. Including the fast tokenizer files and optional training logs, allocate **≈ 500 MB** of disk space.

Performance characteristics: Using the AutoModel with torchscript or ONNX export can further reduce latency to **≈ 5 ms** per sentence on a V100 for a batch size of 8. The fast tokenizer adds negligible overhead (< 1 ms per batch).

Use Cases

HerBERT’s strong Polish language understanding makes it ideal for:

• Customer‑support chatbots that need to parse user queries, detect intent, and extract entities in Polish.
• Sentiment monitoring for social‑media streams, product reviews, and news articles.
• Legal and compliance text analysis (e.g., contract clause extraction, policy compliance checks).
• Search and recommendation engines that rely on semantic similarity and query expansion.
• Academic research in Polish linguistics, dialectology, and corpus studies.

Real‑world examples include a Polish e‑commerce platform using HerBERT to automatically tag product descriptions, a media monitoring service that classifies news sentiment, and a government agency that extracts named entities from public records. The model integrates seamlessly with Hugging Face pipelines, allowing developers to plug it into existing PyTorch, TensorFlow, or JAX codebases.

Training Details

HerBERT was trained on six large‑scale Polish corpora, totalling **≈ 9 B tokens** and **≈ 30 M documents**:

• CCNet Middle (3.2 B tokens, 7.9 M docs)
• CCNet Head (2.6 B tokens, 7.0 M docs)
• National Corpus of Polish (1.4 B tokens, 3.9 M docs)
• Open Subtitles (1.0 B tokens, 1.1 M docs)
• Wikipedia (260 M tokens, 1.4 M docs)
• Wolne Lektury (41 M tokens, 5.5 k docs)

The tokenizer was built with the CharBPETokenizer from Hugging Face tokenizers, yielding a 50 k subword vocabulary. Training employed **dynamic whole‑word masking** (15 % of tokens per step) and the **Sentence‑Structural Objective**, which predicts sentence‑level syntactic relations, encouraging the model to capture both lexical and structural information.

Training was conducted using the transformers library v2.9, with mixed‑precision (FP16) on a multi‑GPU setup (likely 8 × NVIDIA V100 32 GB). The total compute budget is estimated at **≈ 2 M GPU‑hours**, reflecting the large corpus size and the dual‑objective training.

Fine‑tuning is straightforward: the model can be loaded via AutoModel or AutoModelForSequenceClassification and adapted to any downstream Polish task with a modest amount of labeled data (often < 10 k examples). The cased nature of the model preserves capitalization, which is beneficial for tasks like NER.

Related Papers

The primary research foundation is the ACL‑2021 paper:

• HerBERT: Efficiently Pretrained Transformer‑based Language Model for Polish

The paper details the MLM + SSO training regime, whole‑word masking, and the extensive Polish corpora used. It also presents benchmark results on PolEval and CLARIN tasks, establishing HerBERT as a state‑of‑the‑art Polish language model.

Additional background can be found in the original BERT paper (Devlin et al., 2018) and the multilingual BERT work (Conneau et al., 2020), which inspired the architecture and training objectives.

Licensing Information

The model is released under the **Creative Commons Attribution 4.0 International (CC‑BY‑4.0)** license, as indicated in the README and the model card. This permissive license permits:

• Free use, distribution, and modification of the model weights and code.
• Commercial exploitation, including integration into proprietary products and services.
• Creation of derivative works (e.g., fine‑tuned versions) without additional fees.

The only mandatory condition is **attribution**: any public use must credit the original authors (Allegro ML Research Team and the Linguistic Engineering Group at the Polish Academy of Sciences) and provide a link to the original model card. No “share‑alike” or “non‑commercial” restrictions apply.

Because the license is explicit, you can safely embed HerBERT in SaaS platforms, mobile apps, or on‑premise solutions, provided you keep the attribution notice in your documentation or UI.