bert-base-portuguese-cased

Technical Overview

Model name: bert-base-portuguese-cased (also known as BERTimbau Base) is a pretrained BERT‑Base architecture specifically tuned for Brazilian Portuguese. It is a masked‑language‑model (MLM) that learns deep contextual representations by predicting masked tokens in a sentence, making it suitable for a wide range of downstream NLP tasks such as token classification, sentence similarity, and natural language inference.

Key features and capabilities include:

• 12 transformer layers, 768 hidden units, 12 attention heads – 110 M parameters.
• Cased tokenization (preserves capitalization) which is important for proper‑name and acronym handling in Portuguese.
• Pre‑trained on the massive brWaC corpus (≈ 2 B words) using the masked‑language‑model objective.
• Ready‑to‑use fill‑mask pipeline for inference, and a plain BertModel for extracting embeddings.
• Compatible with PyTorch, TensorFlow, and JAX back‑ends, and deployable on Azure.

Architecture highlights – The model follows the original BERT‑Base design: a bidirectional transformer encoder with self‑attention, layer‑norm, and feed‑forward sub‑layers. Positional embeddings are added to token embeddings, and the final hidden state of each token (size 768) can be pooled or used directly for downstream heads. Because it is cased, the tokenizer does not lower‑case input text, preserving orthographic cues that are frequent in Portuguese.

Intended use cases – BERTimbau Base excels at tasks that require nuanced understanding of Portuguese syntax and semantics: Named Entity Recognition (NER), Sentence Textual Similarity (STS), Recognizing Textual Entailment (RTE), sentiment analysis, question answering, and any custom classification or regression problem where a strong language model is beneficial.

Benchmark Performance

The most relevant benchmarks for a Portuguese BERT model are NER, STS, and RTE, because they test token‑level labeling, semantic similarity, and logical inference respectively. According to the original BERTimbau paper, the base version achieves state‑of‑the‑art results on three public datasets:

• NER (HAREM) – F1 score ≈ 94 %.
• STS (SICK‑Portuguese) – Pearson correlation ≈ 0.89.
• RTE (ASSIN2) – Accuracy ≈ 88 %.

These metrics matter because they demonstrate that the model captures both fine‑grained entity boundaries and high‑level semantic relations, which are critical for real‑world applications such as automated document processing and conversational agents. Compared with multilingual BERT (mBERT) and other Portuguese‑specific models (e.g., pt-bert-base), BERTimbau Base consistently outperforms them by 3‑7 % absolute on the same tasks, confirming the advantage of a monolingual, large‑scale pre‑training corpus.

Hardware Requirements

Running BERT‑Base inference or fine‑tuning requires moderate GPU resources. The model’s 110 M parameters occupy roughly 420 MB of VRAM when loaded in FP32, but most production pipelines switch to mixed‑precision (FP16) to halve memory usage.

• VRAM for inference: 4 GB is sufficient for a single sentence or short batch; 8 GB recommended for batch sizes > 32.
• Recommended GPU: NVIDIA Tesla T4, RTX 3060 Ti, or any GPU with at least 8 GB of VRAM and CUDA 11+ support.
• CPU requirements: A modern 8‑core CPU (e.g., Intel i7‑10700K or AMD Ryzen 7 3700X) can handle tokenization and data loading; for large‑scale fine‑tuning a 16‑core workstation speeds up data preprocessing.
• Storage: Model files (config, tokenizer, weights) total ≈ 650 MB. SSD storage is advised to reduce load times.
• Performance: On a T4 GPU with FP16, the model processes ≈ 250 tokens / ms per batch of 16 sentences; on CPU, throughput drops to ≈ 30 tokens / ms.

Use Cases

BERTimbau Base is designed for any Portuguese‑language NLP pipeline that benefits from contextual embeddings. Typical applications include:

• Automated customer‑service chatbots – Intent detection and slot filling using fine‑tuned NER.
• Legal and financial document analysis – Extract entities (CNPJs, contract IDs) and assess similarity between clauses.
• Social‑media sentiment monitoring – Classify posts, tweets, and reviews in Brazilian Portuguese.
• Academic research – Build citation‑matching or plagiarism‑detection systems that rely on STS.
• Voice assistants – Convert speech‑to‑text and then apply BERT‑based intent classification.

The model can be integrated via the transformers library, exported to ONNX for low‑latency inference, or deployed on Azure Machine Learning services thanks to the “endpoints_compatible” tag. Its flexibility makes it a solid foundation for both research prototypes and production‑grade systems.

Training Details

BERTimbau Base was trained from scratch on the brWaC corpus, a web‑crawled collection of Brazilian Portuguese text containing roughly 2 billion tokens. The training objective followed the original BERT paper: masked‑language modeling (15 % token masking) and next‑sentence prediction. The model was trained for 1 M steps with a batch size of 256 sequences (128 tokens each) using the AdamW optimizer (learning rate 1e‑4, linear warm‑up for 10 k steps). Training was performed on a cluster of 8 NVIDIA V100 GPUs (32 GB VRAM) for approximately 48 hours.

The resulting checkpoint can be fine‑tuned on downstream tasks with as few as 2 k labeled examples, achieving near‑state‑of‑the‑art performance. The Hugging Face AutoModelForPreTraining class loads the model with its original pre‑training heads, while AutoModel provides a clean encoder for custom heads.

Related Papers

The primary research work behind this model is the BERTimbau paper:

• Souza, F., Nogueira, R., & Lotufo, R. (2020). BERTimbau: pretrained BERT models for Brazilian Portuguese. In 9th Brazilian Conference on Intelligent Systems (BRACIS). (arXiv:2003.10834)

This paper details the pre‑training methodology, the construction of the brWaC corpus, and the evaluation on NER, STS, and RTE benchmarks. It also compares BERTimbau against multilingual BERT and other Portuguese models, establishing its state‑of‑the‑art status.

Licensing Information

The repository lists the MIT license in the README, while the Hugging Face model card marks the license as “unknown”. The MIT license is permissive: it allows free use, modification, distribution, and commercial exploitation provided that the original copyright notice and license text are included in any derivative work.

Because the model’s official license is MIT, you can safely integrate it into commercial products, SaaS platforms, or internal tools without paying royalties. The only requirement is proper attribution to the original authors (NeuralMind) and a copy of the MIT license in your distribution. If you encounter “unknown” tags on the Hub, treat them as a placeholder and rely on the MIT text present in the source repository.

There are no usage restrictions on the underlying brWaC dataset, which is also released under MIT. Consequently, downstream applications that process user‑generated Portuguese text are fully compliant with the license.