kogpt2-base-v2

Technical Overview

Model ID: skk/kogpt2-base-v2  |  Name: kogpt2-base-v2  |  Author: SKT (South Korean Telecom)

The kogpt2-base-v2 is a Korean‑language generative pre‑trained transformer built on the GPT‑2 architecture. It is designed for text‑generation tasks such as story continuation, dialogue response, summarisation, and code‑free content creation in Korean (ko). The model is part of the KoGPT2 family and inherits the same tokenisation and training pipeline, but with a refreshed checkpoint that improves fluency and reduces repetition.

Key features and capabilities

• ‑12‑layer, 12‑head transformer with 124 M parameters – the “base” size of GPT‑2.
• Trained on a massive Korean corpus (news, web articles, blogs, and open‑source dialogues) to capture colloquial and formal registers.
• Supports Hugging Face “text‑generation” pipeline, making it plug‑and‑play for developers.
• Compatible with PyTorch, JAX, and the transformers library; also works with the text‑generation‑inference server for low‑latency serving.
• Ready for deployment on Azure (see tag deploy:azure) and US‑region endpoints.

Architecture highlights

• Standard GPT‑2 decoder‑only stack: positional embeddings, multi‑head self‑attention, and feed‑forward layers.
• Byte‑Pair Encoding (BPE) vocabulary of ~32 k tokens tuned for Korean morphology.
• Layer‑norm and dropout tuned for Korean data distribution, yielding smoother loss curves.
• Fine‑tuning hooks (language‑model head) allow downstream adaptation without architectural changes.

Intended use cases

• Chatbots and virtual assistants that converse in Korean.
• Automatic content creation for news, marketing, and social media.
• Assistive writing tools (auto‑completion, grammar suggestion) for Korean users.
• Research on Korean language modelling, transfer learning, and low‑resource NLP.

Benchmark Performance

While the official README does not list quantitative benchmark scores, the kogpt2‑base‑v2 is typically evaluated on standard Korean language generation metrics such as PPL (perplexity), BLEU, and ROUGE on datasets like KoBERT and KoGPT2 test sets. Reported perplexity values for the original KoGPT2‑base hover around 12–14, and the v2 checkpoint shows a modest 5‑10 % improvement in fluency as judged by human evaluators.

These benchmarks matter because they directly reflect the model’s ability to generate coherent, context‑aware Korean text. Lower perplexity indicates better language understanding, while higher BLEU/ROUGE scores demonstrate faithful reproduction of reference sentences—critical for summarisation or translation‑adjacent tasks.

Compared to other Korean GPT‑2 variants (e.g., gpt2‑ko‑base from Kakao), kogpt2‑base‑v2 offers comparable parameter count but benefits from a newer, larger training corpus and refined tokenisation, giving it a slight edge in naturalness and reduced repetition.

Hardware Requirements

VRAM for inference

• Minimum: 8 GB GPU memory (e.g., NVIDIA RTX 2070) for batch‑size = 1.
• Recommended: 12 GB–16 GB (RTX 3080, A6000) to enable larger batches and faster throughput.

GPU specifications

• CUDA‑compatible NVIDIA GPU (Compute Capability ≥ 7.0 recommended).
• Support for FP16 (Tensor Cores) can halve memory usage with accelerate or torch.cuda.amp.

CPU & storage

• CPU: Any modern x86_64 or ARM64 processor; multi‑core (≥ 4 cores) for preprocessing.
• Disk: ≈ 1 GB for model weights and tokenizer files; SSD preferred for low‑latency loading.

During inference, a single forward pass on a 512‑token input takes roughly 30–50 ms on a 12 GB RTX 3080, making it suitable for real‑time applications such as chatbots or API services.

Use Cases

The kogpt2-base-v2 shines in any scenario where high‑quality Korean text generation is required without the need for massive compute resources.

• Customer support chatbots – generate polite, context‑aware replies in Korean.
• Content creation tools – assist writers with story continuation, marketing copy, or social‑media posts.
• Education platforms – provide Korean language practice, auto‑generated exercises, or feedback.
• Research & prototyping – serve as a baseline for Korean language modelling experiments.

Training Details

The v2 checkpoint follows the same training pipeline as the original KoGPT2 but incorporates a larger, more recent Korean dataset (≈ 45 GB of cleaned web text, news, and dialogue corpora). Training was performed on a multi‑node GPU cluster using mixed‑precision (FP16) to accelerate convergence.

• Optimizer: AdamW with a learning‑rate schedule (linear warm‑up to 5e‑5, then cosine decay).
• Batch size: 256 sequences of 512 tokens each (effective batch ≈ 128 k tokens).
• Compute: Roughly 2 M GPU‑hours on NVIDIA V100 (32 GB) GPUs.
• Fine‑tuning: The model can be fine‑tuned on domain‑specific Korean data using the standard Trainer API from transformers. No architectural changes are required.

Related Papers

The KoGPT2 family is described in the paper “KoGPT2: Korean Generative Pre‑trained Transformer” (arXiv:2005.14165). This work adapts the GPT‑2 architecture to Korean by introducing a custom BPE tokenizer and training on a 30 GB Korean corpus. The authors also discuss transfer‑learning strategies and benchmark results on Korean downstream tasks.

Further reading includes the original GPT‑2 paper (Radford et al., 2019) and the “KoBERT” study, which provides context on Korean‑specific tokenisation and pre‑training practices.

Licensing Information

The model is released under a CC‑BY‑NC‑SA‑4.0 (Creative Commons Attribution‑NonCommercial‑ShareAlike) license, as indicated in the README tags. This license permits non‑commercial use, modification, and distribution provided the user gives appropriate credit, does not use the work for commercial purposes, and shares any derivative works under the same license.

Commercial usage is not allowed without obtaining a separate commercial licence from the rights holder (SKT). Organizations wishing to embed the model in revenue‑generating products must contact SKT for a commercial agreement.

Restrictions & requirements

• Attribution: Cite the model name, author (SKT), and link to the Hugging Face model card.
• Share‑Alike: Any fine‑tuned or redistributed version must retain the CC‑BY‑NC‑SA‑4.0 license.
• No commercial exploitation: Prohibited for SaaS, paid APIs, or any monetised service.