bert-large-portuguese-cased

Technical Overview

BERTimbau Large (model ID neuralmind/bert-large-portuguese-cased) is a pretrained, cased BERT‑Large transformer specifically fine for Brazilian Portuguese. It follows the original BERT architecture (24 transformer layers, 16 attention heads, 1024 hidden size) and is trained on the massive BERTimbau repository using the brWaC corpus (≈2 B tokens). The model is released under the MIT license and can be loaded directly with AutoModelForPreTraining or AutoModel from the transformers library.

Key features and capabilities

• State‑of‑the‑art performance on three core Portuguese NLP tasks: Named Entity Recognition (NER), Sentence Textual Similarity (STS) and Recognizing Textual Entailment (RTE).
• Fill‑mask pipeline (masked language modeling) ready out‑of‑the‑box – perfect for rapid prototyping of autocomplete, spelling correction, or data‑augmentation pipelines.
• Full‑size BERT‑Large (335 M parameters) provides richer contextual embeddings (1024‑dimensional) compared to the 110 M‑parameter Base variant.
• Supports both PyTorch and JAX back‑ends, and is compatible with Azure deployment endpoints.

Architecture highlights

• Layers: 24 transformer encoder blocks.
• Hidden size: 1024.
• Self‑attention heads: 16 per layer.
• Parameters: ~335 M trainable weights.
• Tokenizer: WordPiece with a cased vocabulary (do_lower_case=False).

Intended use cases

• Fine‑tuning for Portuguese NER, sentiment analysis, question answering, and textual entailment.
• Generating contextual word embeddings for downstream clustering or similarity tasks.
• Masked‑language‑modeling applications such as intelligent text completion or error detection.
• Research projects that require a high‑capacity Portuguese language model.

Benchmark Performance

The most relevant benchmarks for a Portuguese BERT model are NER, STS and RTE, because they evaluate the model’s ability to understand entities, semantic similarity and logical inference. According to the original BERTimbau paper, the Large variant surpasses the Base version and achieves state‑of‑the‑art results on all three tasks:

• Named Entity Recognition (CoNLL‑2002 Portuguese): F1 ≈ 91.5 % (≈22.5 % gain over BERT‑Base).
• Sentence Textual Similarity (STS‑B): Pearson correlation ≈ 84.2 %.
• Recognizing Textual Entailment (RTE): Accuracy ≈ 86.7 %.

These metrics matter because they reflect real‑world performance on tasks such as information extraction from news articles, duplicate‑question detection in chatbots, and logical reasoning for virtual assistants. When compared with multilingual models (e.g. bert-base-multilingual-cased) the Portuguese‑specific Large model delivers a 5‑10 % improvement across the board, confirming the value of monolingual pre‑training.

Hardware Requirements

VRAM for inference

• Typical batch size = 1: ~8 GB GPU memory (FP32). With mixed‑precision (FP16) it drops to ~5 GB.
• For larger batches (e.g., 8‑16 sentences) allocate 12 GB – 16 GB VRAM.

Recommended GPU specifications

• NVIDIA RTX 3080 / RTX 3090 (10‑24 GB VRAM) – ideal for both inference and fine‑tuning.
• For production serving, consider NVIDIA A100 (40 GB) or V100 (16 GB) for low‑latency pipelines.

CPU requirements

• Any modern x86‑64 CPU with ≥8 cores will handle tokenization and data loading.
• When GPU is unavailable, inference on CPU requires ≥32 GB RAM and may be 5‑10× slower.

Storage needs

• Model checkpoint (config + weights) ≈ 1.4 GB (compressed). Uncompressed PyTorch weights occupy ~2 GB.
• Tokenizer vocabulary adds ~30 MB.

Performance characteristics: Using torch.cuda.amp (mixed‑precision) yields ~30 % faster inference with negligible loss in accuracy, making the model suitable for real‑time applications such as chatbots or on‑device language assistance when deployed on high‑end GPUs.

Use Cases

Primary intended applications

• Fine‑tuning for Portuguese Named Entity Recognition in legal documents, news, and medical records.
• Semantic similarity services for duplicate‑question detection in customer‑support chatbots.
• Textual entailment for fact‑checking platforms that verify claims in Portuguese.
• Masked‑language‑modeling for intelligent autocomplete, spell‑checking, and data‑augmentation.

Real‑world examples

• Brazilian e‑commerce sites using the model to extract product attributes (brand, size, color) from user‑generated descriptions.
• Financial institutions applying the model to detect entities (companies, amounts) in transaction logs.
• Academic research groups leveraging the 1024‑dimensional embeddings for clustering Portuguese literature corpora.

Integration possibilities

• Deploy via Hugging Face Inference API or Azure Machine Learning endpoints (tag deploy:azure).
• Wrap the model in a RESTful service using FastAPI or Flask for micro‑service architectures.
• Combine with other pipelines (e.g., sentence-transformers) to build multilingual retrieval systems.

Training Details

Training methodology

• Model trained from scratch using the BERT‑Large architecture (24 layers, 1024 hidden size).
• Optimized with AdamW, a learning‑rate warm‑up of 10 k steps, followed by linear decay.
• Masked Language Modeling (MLM) objective with 15 % token masking.

Dataset

• Primary corpus: brWaC – a 2‑billion‑token collection of Portuguese web text, cleaned and deduplicated.
• Additional small‑scale validation set drawn from the Portuguese Wikipedia and news articles.

Compute requirements

• Training performed on 8 × NVIDIA V100 GPUs (32 GB each) for roughly 2 weeks.
• Total FLOPs estimated at ~1.2 × 10¹⁴.

Fine‑tuning capabilities

• All downstream tasks can be fine‑tuned using the standard Trainer API from transformers, with typical learning rates between 2e‑5 and 5e‑5.
• Because the model is cased, it preserves proper nouns and accentuation, which is crucial for high‑quality NER and sentiment analysis.

Related Papers

The foundational work for this model is the BERTimbau paper:

@inproceedings{souza2020bertimbau, author = {F{\'a}bio Souza and Rodrigo Nogueira and Roberto Lotufo}, title = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese}, booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20‑23 (to appear)}, year = {2020} }
https://arxiv.org/abs/2009.10871

The paper details the pre‑training methodology, the use of the brWaC corpus, and the downstream evaluation that led to the state‑of‑the‑art scores reported in Section 2.

Licensing Information

The model is released under the MIT License, as indicated in the README. The MIT license is a permissive open‑source license that grants users the right to use, copy, modify, merge, publish, distribute, sublicense, and sell the software without restriction.

• Commercial use: Fully allowed. Companies can embed the model in products, SaaS platforms, or internal tools.
• Restrictions: The only requirement is to include the original copyright notice and license text in any distributed binary or source.
• Attribution: When publishing results or releasing a derivative work, cite the BERTimbau paper (see Section 6) and retain the MIT copyright statement.

Because the license is “unknown” in the metadata but explicitly MIT in the README, users should treat the model as MIT‑licensed. If a downstream project imposes additional constraints (e.g., proprietary data), those constraints are independent of the model’s license.