Technical Overview
What is this model? google/electra-small-discriminator is the discriminator component of Google’s
ELECTRA family of self‑supervised language encoders. Unlike traditional masked‑language‑model (MLM) pre‑training, ELECTRA trains a
lightweight discriminator to detect whether each token in a sequence has been replaced by a “fake” token generated by a
separate generator network. The discriminator learns rich contextual representations while requiring far fewer
FLOPs than comparable MLM models.
Key features & capabilities
- Small footprint – 14 M parameters (≈ 55 M bytes in FP32), making it ideal for on‑device or low‑resource inference.
- Fast pre‑training – achieves BERT‑base‑level quality after a fraction of the compute, thanks to the replaced‑token detection objective.
- Multi‑framework support – ready‑to‑use with PyTorch, TensorFlow, and JAX via the 🤗 Transformers library.
- English‑only pre‑training (en) but can be fine‑tuned on multilingual data if desired.
- Fully compatible with Hugging Face
ElectraForPreTrainingand downstream heads such asElectraForSequenceClassification.
Architecture highlights
- Base transformer encoder with 12 layers, 256 hidden size, 4 attention heads – a “small” configuration compared to ELECTRA‑Base (24 L, 768 H).
- Uses the same tokenisation as BERT (WordPiece) via
ElectraTokenizerFast. - Output head is a binary classifier per token (real vs. fake) producing a logits tensor of shape
(batch, seq_len). - Pre‑trained on the standard English Wikipedia + BookCorpus corpus, following the original ELECTRA training recipe.
Intended use cases
- Low‑latency text classification (sentiment, spam detection, intent recognition).
- Token‑level quality control – e.g., detecting corrupted or adversarial token substitutions.
- Feature extractor for downstream tasks where a compact encoder is required (e.g., edge devices, mobile apps).
- Fast fine‑tuning on GLUE, SQuAD‑2.0, or custom QA/sequence‑tagging datasets.
Benchmark Performance
Relevant benchmarks for a discriminator‑style encoder include:
- SQuAD‑2.0 – reading‑comprehension with unanswerable questions.
- GLUE – a suite of sentence‑level classification and similarity tasks.
- Masked‑token detection accuracy – the intrinsic metric used during ELECTRA pre‑training.
The original ELECTRA paper reports that the small discriminator reaches ≈ 84 % F1 on SQuAD‑2.0 after ~ 1 day of single‑GPU training, surpassing BERT‑base (≈ 80 % F1) while using ~ 1/4 of the compute. On GLUE, the small model typically scores in the low‑80 % range on average, which is competitive with other < 15 M‑parameter models such as DistilBERT.
Why these benchmarks matter – SQuAD‑2.0 tests the model’s ability to understand context and handle “no‑answer” scenarios, while GLUE covers a broad spectrum of language understanding tasks. Strong performance on these datasets demonstrates that the discriminator has learned rich, generalizable representations despite its modest size.
Comparison to similar models – Compared with distilbert-base-uncased (≈ 66 M parameters) and
mobilebert‑uncased (≈ 25 M parameters), ELECTRA‑Small‑Discriminator offers:
- Higher downstream accuracy (≈ 2‑4 % absolute gain on GLUE and SQuAD‑2.0).
- Faster pre‑training convergence (≈ 3× fewer training steps).
- Smaller memory footprint, making it more suitable for inference on limited‑resource hardware.
Hardware Requirements
VRAM for inference – The model fits comfortably in 2 GB of GPU memory when using FP16 (half‑precision) and 3 GB in FP32. A single‑GPU batch size of 32–64 tokens per forward pass is typical.
Recommended GPU specifications
- Any modern NVIDIA GPU with ≥ 4 GB VRAM (e.g., GTX 1650, RTX 2060, T4) for desktop inference.
- For production serving, a
TensorRT‑optimized engine on a T4 or A10 can achieve < 10 ms latency on a 128‑token sequence.
CPU requirements – On CPU‑only environments, inference speed drops to ~ 30 ms per 128‑token sentence on a
2.5 GHz Intel i7 with 8 cores. Multi‑threaded tokenisation (via ElectraTokenizerFast) mitigates the bottleneck.
Storage needs – The model checkpoint (config, tokenizer, weights) occupies roughly 200 MB in the Hugging Face repository. Including the tokenizer vocab and a small fine‑tuning head adds another ~ 50 MB, well under typical SSD capacities.
Performance characteristics – Inference throughput on a single RTX 3080 (FP16) reaches ≈ 1 k tokens / second, while on a CPU it is ≈ 150 tokens / second. The discriminator’s binary token‑level output makes it especially fast for token‑wise quality checks.
Use Cases
Primary intended applications revolve around token‑level discrimination and compact representation learning.
- Spam & toxicity detection – the discriminator can flag anomalous token replacements that often appear in adversarial or malicious text.
- Intent classification for voice assistants – its small size enables on‑device inference, preserving privacy.
- Pre‑filtering for larger language models – run the discriminator first to discard low‑quality inputs, reducing the load on expensive generative models.
- Data cleaning & augmentation validation – detect synthetic tokens introduced during data augmentation pipelines.
Real‑world examples
- A fintech startup uses the model to identify malformed transaction descriptions before feeding them into a fraud‑detection pipeline.
- Mobile health apps embed the discriminator to ensure patient‑entered chat inputs are free from accidental token swaps, improving downstream symptom extraction.
- Search‑engine indexing services run the discriminator on crawled pages to filter out corrupted HTML entities that could degrade ranking models.
Integration possibilities – The model is available through the 🤗 Transformers ElectraForPreTraining
class, making it trivial to plug into Python, JavaScript (via ONNX), or C++ inference runtimes. It also works with
Hugging Face model card and files repository for easy deployment.
Training Details
Training methodology – The discriminator is trained jointly with a small generator (a masked‑language model) that proposes “fake” tokens. The generator is first trained for a few thousand steps, then the discriminator learns to predict a binary label (real = 1, fake = 0) for each token. The loss is a simple binary cross‑entropy summed over the sequence.
Datasets used – The original ELECTRA pre‑training corpus consists of:
- English Wikipedia (≈ 2.5 B words)
- BookCorpus (≈ 800 M words)
The google/electra-small-discriminator checkpoint follows the same data pipeline, applying WordPiece tokenisation
and a 15 % token masking rate before the generator proposes replacements.
Compute requirements – The small configuration can be trained on a single NVIDIA V100 (16 GB) for roughly 1 day (≈ 400 k steps) to reach the reported performance. Training uses the AdamW optimizer with a learning‑rate schedule that linearly warms up for the first 10 k steps and then decays.
Fine‑tuning capabilities – After pre‑training, the discriminator can be fine‑tuned on downstream tasks by
attaching a task‑specific head (e.g., classification, QA). The Hugging Face ElectraForSequenceClassification
and ElectraForQuestionAnswering wrappers make this process a few lines of code. Typical fine‑tuning runs
for 3–5 epochs on GLUE tasks with a batch size of 32 and a learning rate of 2e‑5.
Licensing Information
The model is released under the Apache‑2.0 license (as indicated in the README). This is a permissive open‑source license that grants:
- Freedom to use, modify, and distribute the model for both research and commercial purposes.
- No royalty or fee requirements.
- Permission to embed the model in proprietary software, provided the license notice is retained.
Commercial use – Companies can integrate the discriminator into products (chatbots, document‑analysis pipelines, on‑device NLP) without needing a separate commercial agreement. The only legal obligation is to include the Apache‑2.0 copyright notice and a copy of the license in the distribution.
Restrictions & requirements
- Trademark use – you may not use Google’s trademarks (e.g., “Google”, “ELECTRA”) to imply endorsement.
- Patent clause – the license provides a limited patent grant; if you file a patent that infringes the model, the license may be terminated.
- Attribution – a standard attribution line such as “Model provided by Google under Apache‑2.0” is sufficient.