camembert-base

Technical Overview

CamemBERT‑base (model ID almanach/camembert-base) is a French‑language masked‑language‑model built on the RoBERTa architecture. It is designed to understand and generate French text by predicting masked tokens (the fill‑mask pipeline) and by providing contextual embeddings for downstream tasks such as classification, named‑entity recognition, and semantic search.

Key features and capabilities:

• 110 M parameters – a “base‑size” model that balances accuracy and efficiency.
• Pre‑trained on OSCAR (138 GB of French text) – a massive, diverse web‑crawled corpus that gives the model a broad linguistic coverage.
• Sub‑word tokenizer (SentencePiece) – handles French morphology and rare words without exploding the vocabulary size.
• Mask‑filling (fill‑mask) support – ready‑to‑use via the Hugging Face pipeline for zero‑shot cloze tasks.
• Hidden‑state extraction – all 13 layers (input embedding + 12 transformer blocks) can be returned for fine‑grained feature analysis.

Architecture highlights:

• Based on the RoBERTa‑base transformer (12 layers, 12 attention heads, 768 hidden size).
• Trained with the same masked‑language‑model objective as RoBERTa (random token masking, next‑sentence prediction disabled).
• Uses the CamembertTokenizer which is a French‑specific SentencePiece model.

Intended use cases include:

• French‑language masked token prediction (e.g., “Le camembert est un fromage de !”).
• Feature extraction for downstream NLP tasks (sentiment analysis, topic classification, NER, etc.).
• Research on French language representation and transfer learning.

Benchmark Performance

CamemBERT‑base has been evaluated on the original CamemBERT paper (arXiv:1911.03894) where it achieved state‑of‑the‑art results on French benchmarks such as GLUE‑FR and XNLI‑FR. Reported scores include:

• ~84 % accuracy on the French MLM (masked‑language‑model) validation set.
• ~90 % F1 on French NER tasks, surpassing multilingual BERT baselines by several points.

These benchmarks matter because they test a model’s ability to understand context, resolve ambiguities, and transfer knowledge to downstream tasks. Compared to the multilingual bert-base-multilingual-cased (≈110 M parameters, 104 languages), CamemBERT‑base consistently outperforms on French‑only tasks, confirming the advantage of monolingual pre‑training on a large, high‑quality French corpus.

Hardware Requirements

For inference with the default CamembertModel (output hidden size = 768):

• VRAM: ~2 GB for a single‑sentence batch (batch size = 1, sequence length ≤ 512).
• Recommended GPU: NVIDIA RTX 3060 or higher (8 GB VRAM) for comfortable batch processing.
• CPU: Modern multi‑core CPU (e.g., Intel i7‑9700K) can run inference at ~30‑50 ms per sentence when GPU is unavailable, though latency will be higher.
• Storage: Model files (weights + tokenizer) occupy ~420 MB (safetensors format).
• Performance: Throughput of ~200‑300 tokens / second on a mid‑range GPU; higher batch sizes scale linearly up to VRAM limits.

Use Cases

CamemBERT‑base is especially suited for French‑centric applications:

• Chatbots & virtual assistants – generate or complete French sentences in real time.
• Content moderation – detect inappropriate language or policy violations using contextual embeddings.
• Document classification – fine‑tune on French news, legal, or medical corpora for topic tagging.
• Search & recommendation – embed queries and documents for semantic matching in French e‑commerce platforms.
• Academic research – study French language phenomena, dialectal variation, or cross‑lingual transfer.

Training Details

CamemBERT‑base was trained on the OSCAR dataset, a 138 GB French‑language crawl of the Common Crawl. Training followed the RoBERTa “large‑scale” recipe:

• Batch size: 8 k tokens per GPU (gradient accumulation to reach effective batch size of 32 k).
• Learning rate: 1e‑4 with linear warm‑up (10 % of total steps) and cosine decay.
• Training steps: ~500 k (≈3 days on 8 × NVIDIA V100 32 GB GPUs).
• Optimizer: AdamW with weight decay = 0.01.
• Mixed‑precision (FP16) training to reduce memory footprint.

The model is released with output_hidden_states=True optional, enabling extraction of all layer embeddings for fine‑grained analysis. Fine‑tuning for downstream tasks can be performed with the standard Hugging Face Trainer API, typically requiring only a few epochs on a modest French dataset (e.g., 10 k labeled examples).

Related Papers

The primary research foundation is the CamemBERT paper:

• CamemBERT: a Tasty French Language Model (arXiv:1911.03894)

This work adapts the RoBERTa training regime to French, demonstrating that monolingual pre‑training on a large web‑crawled corpus (OSCAR) yields significant gains over multilingual alternatives. The paper also details the masked‑language‑model objective, tokenization strategy, and extensive benchmark evaluation.

Licensing Information

The model card lists the MIT license (the license field in the README). The MIT license is permissive:

• Allows commercial and non‑commercial use without fee.
• Permits modification, distribution, and private use.
• Requires that the original copyright notice and license text be included in any redistributed copies.

Because the license is MIT, there are no restrictions on integrating CamemBERT‑base into proprietary products, provided the attribution notice is retained. If you plan to redistribute the model (e.g., on a hardware device), include the MIT license text alongside the model files.