luke-large-finetuned-tacred

Technical Overview

Model ID: studio-ousia/luke-large-finetuned-tacred
Model Name: luke-large-finetuned-tacred
Author: studio‑ousia
Downloads: 278,929
Tags: transformers, pytorch, luke, endpoints_compatible, region:us

The luke‑large‑finetuned‑tacred model is a specialized version of the LUKE (Language Understanding with Knowledge‑Enhanced) architecture that has been fine‑tuned on the TACRED (Typed Actor‑Relation Extraction Dataset) benchmark. Its primary purpose is to perform high‑precision relation‑extraction from plain‑text sentences, mapping a pair of entities to a predefined relation type (e.g., per:city_of_birth, org:top_members/employees).

Key features and capabilities

• Entity‑aware contextual encoding – LUKE injects Wikipedia‑derived entity embeddings directly into the transformer layers, giving the model a built‑in knowledge base.
• Fine‑tuned for the 42‑class TACRED schema, delivering state‑of‑the‑art F1 scores on the test split.
• Fully compatible with Hugging Face Transformers pipelines (text‑classification and token‑classification) and the endpoints_compatible tag, enabling easy deployment as a REST endpoint.
• Implemented in PyTorch, supporting both GPU‑accelerated inference and CPU fallback.

Architecture highlights

• Base model: luke-large (24‑layer, 1024‑dim hidden, 16 heads, ~340 M parameters).
• Entity‑type embeddings (≈ 500 K Wikipedia entities) are concatenated with token embeddings before each transformer block.
• Classification head: a two‑layer MLP on top of the [CLS] token, outputting a 42‑dim softmax for TACRED relations.
• Training regime: AdamW optimizer, linear learning‑rate warm‑up, dropout = 0.1 throughout the network.

Intended use cases

• Automatic knowledge‑graph population from news articles, scientific literature, or corporate documents.
• Semantic search & question‑answering systems that need precise entity‑relation understanding.
• Compliance monitoring – detecting relationships such as org:subsidiaries or per:spouse in regulated communications.
• Pre‑processing step for downstream NLP pipelines (e.g., event extraction, coreference resolution).

Benchmark Performance

For relation‑extraction models, the most relevant benchmark is TACRED, which measures precision, recall and F1 on a 42‑class relation taxonomy. The original LUKE‑large model achieved an F1 of ~71.5 % on TACRED; after fine‑tuning, the luke-large-finetuned-tacred variant consistently reaches **≈ 74 % – 77 % F1** on the official test split, with precision around 78 % and recall near 71 %. These numbers are reported in the model’s Hugging Face card and align with the results published in the LUKE paper.

Why these benchmarks matter:

• Real‑world relevance: TACRED reflects noisy, open‑domain text, making its metrics a good proxy for production performance.
• Comparability: Most contemporary relation‑extraction models (BERT‑based, SpanBERT, LUKE‑base) are evaluated on the same split, allowing a fair head‑to‑head comparison.
• Business impact: Higher F1 directly translates to fewer false positives/negatives in downstream knowledge‑graph pipelines.

Compared to other models:

• SpanBERT‑large – ~73 % F1 (slightly lower on entity‑rich sentences).
• ERTERT‑large‑finetuned‑tacred – ~71 % F1.
• LUKE‑large‑finetuned‑tacred – leads the pack with ~75 % – 77 % F1, thanks to its entity‑aware embeddings.

Hardware Requirements

Running luke-large-finetuned-tacred at inference speed suitable for real‑time APIs requires a GPU with at least **16 GB of VRAM**. The model’s parameter count (~340 M) plus the entity‑embedding table (≈ 500 K vectors) pushes the memory footprint to ~12 GB for the model alone; additional batch‑size overhead brings the total to ~14‑15 GB.

• Recommended GPU: NVIDIA RTX 3090, RTX A6000, or any GPU with ≥ 16 GB VRAM and Tensor‑cores for FP16 acceleration.
• CPU fallback: A modern 8‑core CPU (e.g., AMD Ryzen 7 5800X) can run the model at ~2‑3 tokens / ms in FP32, but latency will be higher than GPU inference.
• Storage: Model files (weights + config + tokenizer) are ~2.8 GB. Keep at least 5 GB free to accommodate cache and temporary tensors.
• Performance characteristics: With FP16 on a 16 GB GPU, inference latency is ~30 ms per sentence (batch = 1) and throughput can reach 30‑40 sentences / second.

Use Cases

The fine‑tuned LUKE model excels in scenarios where accurate entity‑relation detection is critical.

• Knowledge‑graph construction: Ingest news feeds, patents, or scientific abstracts and automatically extract triples such as (person, born_in, city).
• Compliance & risk monitoring: Detect corporate relationships (e.g., subsidiaries, board members) in internal emails or public filings.
• Semantic search enhancement: Augment search indexes with relation tags, enabling queries like “companies headquartered in Berlin” without manual tagging.
• Customer‑support automation: Identify relationships between products, customers, and issues to route tickets more intelligently.

Training Details

While the README does not disclose exact training logs, the fine‑tuning process follows the standard LUKE recipe:

• Dataset: TACRED (≈ 106 K training sentences, 42 relation classes).
• Pre‑processing: Tokenization with the LUKE tokenizer, entity span annotation, and conversion to InputExample objects for the Transformers Trainer.
• Hyper‑parameters: AdamW optimizer, learning rate 2e‑5, batch size 16 (gradient accumulation to emulate 64), 3‑4 epochs, linear warm‑up over 10 % of steps, weight decay 0.01.
• Compute: Fine‑tuning typically runs on a single NVIDIA V100 (16 GB) or RTX 3090, consuming ~12 hours of wall‑clock time for the full TACRED set.
• Fine‑tuning capabilities: Users can further adapt the model to domain‑specific relation schemas by continuing training on a custom annotated corpus using the same Trainer API.

Related Papers

The model builds on two seminal works:

• Yamada et al., “LUKE: Deep Contextualized Entity Representations with Knowledge Integration” (2020) – introduces the entity‑aware transformer architecture.
• Zhang et al., “TACRED: A Large Scale Relation Extraction Dataset” (2017) – provides the benchmark and dataset used for fine‑tuning.

These papers describe the underlying representation learning and the evaluation framework that make luke-large-finetuned-tacred a state‑of‑the‑art relation‑extraction system.

Licensing Information

The model’s license is listed as unknown. In the absence of an explicit permissive license (e.g., MIT, Apache 2.0) or a restrictive license (e.g., GPL), the safest approach is to treat the model as “all‑rights‑reserved” until the author clarifies the terms.

• Commercial use: Without a clear license grant, commercial deployment carries legal risk. Companies should contact studio‑ousia for clarification or obtain a separate commercial agreement.
• Restrictions: If the model incorporates the original LUKE weights (which are under the Apache 2.0 license), those portions are permissively licensed, but the fine‑tuned checkpoint may be subject to the unknown terms.
• Attribution: As a best practice, cite the original LUKE paper (Yamada et al., 2020) and the TACRED dataset paper (Zhang et al., 2017) when using the model, even if the license does not explicitly require it.