pegasus-xsum

Technical Overview

Model ID: google/pegasus-xsum
Model Name: Pegasus‑XSum
Author: Google (maintained by @sshleifer)

Pegasus‑XSum is a text‑to‑text transformer that specializes in abstractive summarization. It takes a long article (or any piece of natural‑language text) as input and generates a concise, fluent summary that captures the most important information. The model is built on the Pegasus architecture, which was designed specifically for summarization by treating the task as a “gap‑sentence” generation problem: during pre‑training the model learns to predict masked sentences that are most informative for the whole document.

• Key Features & Capabilities
  • Trained on a mixture of the massive C4 corpus and the news‑oriented HugeNews dataset, giving it a strong grasp of both web‑scale language and newsroom style.
  • Uses a stochastic “gap‑sentence” ratio (15 %‑45 %) and adds 20 % uniform noise to importance scores, which improves robustness to diverse document structures.
  • SentencePiece tokenizer that preserves newline characters, enabling better paragraph‑level segmentation.
  • Supports the Hugging Face summarization pipeline out‑of‑the‑box for both PyTorch and TensorFlow (and JAX) back‑ends.
• Architecture Highlights
  • Encoder‑decoder transformer with 12 encoder layers and 12 decoder layers (the same configuration as Pegasus‑large).
  • Each layer contains 16 attention heads and a hidden size of 1024, totaling roughly 568 M parameters.
  • Pre‑training objective: gap‑sentence generation – the model learns to reconstruct a subset of “important” sentences that have been removed from the source.
  • Fine‑tuned on the XSum dataset (BBC‑style news articles) to produce one‑sentence summaries that are highly abstractive.
• Intended Use Cases
  • Automatic news headline generation.
  • Summarizing long reports, scientific abstracts, or meeting transcripts into a single concise sentence.
  • Pre‑processing for downstream tasks such as question answering, where a short summary can reduce context length.

Benchmark Performance

Summarization models are typically evaluated with ROUGE scores, which compare n‑gram overlap between the generated summary and a human reference. Pegasus‑XSum has been benchmarked on three widely used datasets:

• Samsum (train split)
  • ROUGE‑1: 21.81
  • ROUGE‑2: 4.25
  • ROUGE‑L: 17.45
  • ROUGE‑LSUM: 18.89
  • Average generated length: 20.3 tokens
• XSum (test split)
  • ROUGE‑1: 46.86
  • ROUGE‑2: 24.45
  • ROUGE‑L: 39.05
  • ROUGE‑LSUM: 39.10
  • Average generated length: 22.9 tokens
• CNN/DailyMail (test split)
  • ROUGE‑1: 22.21
  • ROUGE‑2: 7.67
  • ROUGE‑L: 15.40
  • ROUGE‑LSUM: 19.22
  • Average generated length: 25.0 tokens

These benchmarks matter because they reflect how well the model can condense information while preserving factual content. The XSum results (≈ 47 % ROUGE‑1) are competitive with other state‑of‑the‑art summarizers such as BART‑large and T5‑large, especially given the model’s focus on a single‑sentence output. The mixed‑and‑stochastic pre‑training strategy also yields consistent improvements across diverse datasets (see the “Mixed & Stochastic” table in the README).

Hardware Requirements

• VRAM for Inference
  • Full‑precision (FP32) inference typically needs 8 GB–10 GB of GPU memory.
  • Using half‑precision (FP16) or INT8 quantization can reduce the requirement to 5 GB–6 GB.
• Recommended GPU
  • Any NVIDIA GPU with ≥ 8 GB VRAM (e.g., RTX 2070, RTX 3060, A100 40 GB for batch processing).
  • For large‑scale batch inference, consider GPUs with ≥ 16 GB VRAM (e.g., RTX 3080, V100, A100 80 GB).
• CPU Requirements
  • Modern multi‑core CPUs (≥ 8 cores) can handle tokenization and model loading, but GPU acceleration is strongly recommended for low latency.
  • Minimum RAM: 16 GB to hold the model weights and a few concurrent requests.
• Storage Needs
  • Model checkpoint size: roughly 2.5 GB (including tokenizer files).
  • Additional disk space for tokenized caches or fine‑tuning data is optional.
• Performance Characteristics
  • Typical latency for a 1‑k token article on a single RTX 3060 (FP16) ≈ 150 ms.
  • Throughput scales linearly with batch size and GPU memory; a 32‑sample batch on an A100 can process > 200 samples/s.

Use Cases

• News Media & Publishing
  • Automatic headline generation for breaking‑news articles.
  • One‑sentence abstracts for newsletters or RSS feeds.
• Enterprise Knowledge Management
  • Summarizing long internal reports, policy documents, or meeting minutes into a single actionable sentence.
  • Providing quick previews in document search results.
• Research & Academia
  • Generating concise summaries of scientific papers (e.g., arXiv abstracts) to aid literature reviews.
  • Creating short “tweet‑style” summaries for outreach.
• Customer Support & Chatbots
  • Condensing lengthy support tickets or chat logs into a brief summary for agents.

Training Details

Methodology: Pegasus‑XSum follows the “mixed & stochastic” pre‑training regime described in the README. The model is first trained on a blended corpus of C4 (a web‑scale text dataset) and HugeNews (a large news collection). The mixture is weighted by the number of examples in each dataset.

• Training steps: 1.5 M updates (vs. 500 k in the original Pegasus‑large).
• Gap‑sentence ratio: sampled uniformly between 15 % and 45 %.
• Importance‑sentence noise: 20 % uniform perturbation to the importance scores.
• Tokenizer: SentencePiece with newline support, preserving paragraph boundaries.

Datasets used for fine‑tuning:

• XSum – BBC‑style news articles with one‑sentence summaries (the primary fine‑tuning target).
• Additional evaluations on Samsum, CNN/DailyMail, and a range of other summarization benchmarks (see the “Mixed & Stochastic” table).

Compute: The original Pegasus‑large was trained on Google’s TPU‑v3 pods; the mixed‑and‑stochastic variant likely required a similar scale (tens of thousands of TPU‑v3 cores or equivalent GPU clusters). Exact FLOPs are not disclosed, but the 1.5 M‑step schedule suggests a multi‑week training run on a large distributed system.

Fine‑Tuning: The model is fully compatible with Hugging Face’s Trainer API. Users can further fine‑tune on domain‑specific summarization data (e.g., legal contracts, scientific abstracts) by loading the checkpoint and training for a few epochs with a modest learning rate (≈ 5e‑5).

Related Papers

Key research that underpins Pegasus‑XSum includes:

• Pegasus: Pre‑Training with Gap‑Sentence Generation for Abstractive Summarization (Zhang, Zhao, Saleh, Liu, 2019) – the original architecture and pre‑training methodology.
• Additional evaluation tables in the README reference mixed‑and‑stochastic checkpoint experiments, which are extensions of the original Pegasus paper.

Licensing Information

The model card lists the license as unknown. In practice, this means the repository does not ship an explicit permissive license (e.g., MIT, Apache 2.0) or a restrictive one (e.g., GPL). When a license is not declared:

• It is safest to treat the model as all‑rights‑reserved until you obtain clarification from the author or the hosting platform.
• Commercial use is technically not guaranteed; many Google‑origin models are released under permissive terms, but you must verify the specific licensing file (often LICENSE or NOTICE) in the repository.
• Typical requirements for “unknown” models include:
  • Attribution to the original authors (Google, Jingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. Liu).
  • No redistribution of the model weights in a modified form without explicit permission.

Before deploying Pegasus‑XSum in a commercial product, we recommend contacting the model maintainer (@sshleifer) or checking the Hugging Face discussion board for any updates on licensing.