kf-deberta-base

Technical Overview

Model ID: kakaobank/kf-deberta-base  |  Model Name: kf-deberta-base
Author: KakaoBank & FNGuide  |  Language: Korean (ko)
Pipeline Tag: fill‑mask

What is this model? KF‑DeBERTa is a Korean‑language transformer that has been pre‑trained on a mixture of general‑domain corpora and large‑scale financial‑domain text. It is designed to excel both on typical Korean NLP tasks (sentiment analysis, NER, QA, etc.) and on finance‑specific downstream applications such as credit‑risk classification, financial news sentiment, and regulatory document parsing.

Key features & capabilities

• Based on Microsoft’s DeBERTa‑v2 architecture, which improves token‑level disentangling and relative positioning.
• Training objective: Masked Language Modeling (MLM) with the “RTD” (replaced token detection) variant of ELECTRA was evaluated, but the final model kept the DeBERTa‑v2 MLM objective because it yielded higher scores on Korean benchmarks.
• Domain‑adaptive pre‑training: The model first learns from a broad Korean corpus, then continues training on a curated financial corpus, giving it a strong “financial‑aware” language understanding.
• Supports the fill‑mask pipeline out‑of‑the‑box, making it ready for masked‑token prediction, cloze‑style QA, and token‑level inference without additional heads.

Architecture highlights

• Base‑size transformer (≈110 M parameters) with 12 layers, 768 hidden size, 12 attention heads – the same scale as BERT‑Base.
• DeBERTa‑v2 introduces relative position bias and disentangled attention, which improve handling of long‑range dependencies common in financial documents.
• LayerNorm is applied before the attention sub‑layer (pre‑norm), leading to more stable training on large corpora.

Intended use cases

• Financial sentiment analysis on news, reports, or social media.
• Entity extraction for Korean‑language financial NER (e.g., company names, ticker symbols).
• Masked‑language modeling for data augmentation in finance‑specific NLP pipelines.
• General Korean NLP tasks where a high‑quality base model is required (e.g., chat‑bots, document summarization).

Benchmark Performance

KF‑DeBERTa was evaluated on the KLUE benchmark (Korean Language Understanding Evaluation) and a suite of finance‑specific tasks. The model consistently ranks among the top base‑size models, often surpassing larger RoBERTa‑Large and XLM‑R‑Large baselines.

KLUE results (average score 82.83)

Financial‑domain benchmarks (partial table shown) demonstrate strong accuracy on sentiment (FN‑Sentiment v1/v2 > 87 % ACC) and high F1 on financial NER, confirming the model’s domain adaptation.

These benchmarks matter because Korean‑language models often lag behind English counterparts, and finance‑specific tasks demand both linguistic nuance and domain knowledge. KF‑DeBERTa’s superior scores indicate it can be deployed with confidence in production‑grade Korean financial NLP pipelines.

Hardware Requirements

Inference VRAM – The base model (~110 M parameters) comfortably fits within a single 8 GB GPU when using half‑precision (FP16) inference. For full‑precision (FP32) a 12 GB GPU is recommended.

• Recommended GPUs: NVIDIA RTX 3060 (12 GB), RTX 3070/3080 (10‑12 GB), or any GPU with ≥ 8 GB VRAM supporting CUDA 11+.
• CPU inference: A modern 8‑core CPU (e.g., Intel i7‑10700K or AMD Ryzen 7 3700X) can run the model at ~30‑50 tokens/s in FP32; using ONNX Runtime with INT8 quantization can reduce latency substantially.
• Storage: Model files (config, tokenizer, weights) total ~420 MB. Keep at least 1 GB free for caching and temporary files.
• Batch size: For GPU inference, batch sizes of 16‑32 are typical; larger batches may exceed VRAM.

Use Cases

KF‑DeBERTa shines in any Korean‑language scenario where domain‑specific knowledge adds value. Typical applications include:

• Financial sentiment monitoring: Real‑time analysis of news, earnings calls, and social media to gauge market mood.
• Regulatory document processing: Automated extraction of entities (company names, legal citations) and classification of sections in Korean financial regulations.
• Customer support chatbots: Understanding user queries about banking products, with a language model that already knows finance terminology.
• Risk assessment: Scoring loan applications or credit reports using masked‑language predictions to infer hidden risk factors.
• Content recommendation: Personalizing financial newsletters by matching user interests with extracted topics.

Training Details

The authors combined a large, diverse Korean corpus with a specialized financial text corpus. While exact dataset sizes are not disclosed, the following can be inferred:

• Pre‑training objective: Masked Language Modeling (MLM) using the DeBERTa‑v2 loss; an ELECTRA‑style RTD trial was performed but later abandoned.
• Domain‑adaptive stage: After generic pre‑training, the model continued training on a financial‑domain dataset to capture terminology, numeric expressions, and regulatory language.
• Hyper‑parameters (baseline search): batch size {16, 32}, learning rates {1e‑5, 3e‑5, 5e‑5}, weight decay {0, 0.01}, warm‑up proportion {0, 0.1}.
• Compute: Training was performed on multi‑GPU (likely 8‑GPU) NVIDIA A100 or V100 clusters, typical for a ~110 M‑parameter model (≈ 200 K steps).
• Fine‑tuning: The model can be fine‑tuned on any Korean downstream task with the standard Trainer API from 🤗 Transformers. The provided fill‑mask pipeline works out‑of‑the‑box.

Related Papers

KF‑DeBERTa builds on the DeBERTa‑v2 architecture, described in the paper:

• “DeBERTa: Decoding-enhanced BERT with Disentangled Attention” (Microsoft, 2021) – introduces disentangled attention and relative position bias.

The model also benefits from research on masked‑language modeling and domain‑adaptive pre‑training, such as:

• “ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators” – informs the RTD experiment mentioned in the README.

Licensing Information

The README lists the MIT License as the model’s license, even though the tag field shows “unknown”. Under MIT:

• All users may use, copy, modify, merge, publish, distribute, sublicense, and/or sell the software.
• Only the original copyright notice and license text need to be retained in any redistribution.
• There are no commercial restrictions – the model can be integrated into commercial products, SaaS platforms, or internal tools.
• Typical attribution: “Model: kakaobank/kf-deberta-base – © KakaoBank & FNGuide, licensed under MIT.”

If you plan to redistribute the model (e.g., as part of a larger package), keep the MIT license file alongside the model files to stay compliant.