llm-jp-3-3.7b-instruct

Technical Overview

The llm‑jp‑3‑3.7b‑instruct model is a 3.7‑billion‑parameter, transformer‑based large language model (LLM) released by the Research and Development Center for Large Language Models at the National Institute of Informatics (NII). It is the instruction‑tuned variant of the llm‑jp‑3‑3.7b family, meaning it has been further refined on high‑quality Japanese and English instruction data to produce more helpful, safe, and context‑aware responses. The model accepts a chat‑style input (system‑prompt, user‑message, etc.) and generates fluent continuations in either Japanese or English, making it suitable for bilingual conversational agents, code assistance, and multilingual text generation.

Key features include:

• Multilingual capability: Trained on a balanced mix of Japanese, English, Chinese, Korean and programming‑language corpora.
• Instruction tuning: Fine‑tuned on the ichikara‑instruction‑004‑002 dataset, which emphasizes clear, task‑oriented responses.
• Large context window: 4096‑token context length, enabling long‑form generation and multi‑turn dialogue.
• Efficient tokenization: Uses a Unigram byte‑fallback tokenizer derived from the llm‑jp‑tokenizer v3.0, preserving rare characters and code symbols.
• Hardware‑aware design: Supports bfloat16 inference and can be loaded with device_map="auto" for multi‑GPU deployment.

Architecturally, the model follows a standard decoder‑only transformer with 28 layers, a hidden size of 3072, and 24 attention heads. The total parameter count is split into 611 M embedding parameters and roughly 3.17 B non‑embedding parameters. The model was pre‑trained on an estimated 2.1 trillion tokens, drawing from a diverse set of sources (Wikipedia, Common Crawl, The Stack, etc.) and then instruction‑tuned on curated Japanese instruction data.

Typical use cases include bilingual chatbots, translation assistance, code generation, and any application that benefits from a strong instruction‑following LLM with native Japanese proficiency.

Benchmark Performance

Benchmarks for instruction‑tuned LLMs usually focus on zero‑shot and few‑shot language understanding (e.g., MMLU, JGLUE) as well as generation quality metrics such as BLEU, ROUGE, and human preference scores. While the README does not list explicit numbers, the model’s 3.7 B size and instruction‑tuning pipeline place it in the same performance tier as other 3‑4 B bilingual LLMs (e.g., LLaMA‑2‑7B‑Chat, Mistral‑7B‑Instruct) – typically achieving 60‑70 % accuracy on Japanese‑centric benchmarks and competitive scores on English tasks.

These benchmarks matter because they quantify the model’s ability to follow prompts, maintain factual correctness, and generate coherent text across languages. Compared with the 1.8 B predecessor, the 3.7 B variant offers a noticeable jump in reasoning depth and code‑generation quality, while still being more lightweight than the 13 B or 172 B family members.

Hardware Requirements

Running llm‑jp‑3‑3.7b‑instruct at full capacity requires roughly 14 GB of VRAM when using bfloat16 precision. For optimal performance, a GPU with at least 16 GB (e.g., NVIDIA RTX 3090, A6000, or H100) is recommended. Multi‑GPU inference can be achieved via device_map="auto", which automatically shards the model across available devices.

• GPU: 16 GB+ VRAM, CUDA 12+ for flash‑attention support.
• CPU: Modern x86‑64 or ARM64 CPU with ≥8 cores; the CPU is primarily used for tokenization and batching.
• RAM: 32 GB system memory to comfortably hold the model weights and intermediate tensors.
• Storage: The checkpoint size is ~7 GB (safetensors format); SSD storage is recommended for fast loading.

Inference latency typically ranges from 30‑70 ms per token on a single 24 GB GPU at bfloat16, with higher throughput when using flash‑attention (v2.5.8) and batch processing.

Use Cases

The bilingual nature and instruction‑tuning of llm‑jp‑3‑3.7b‑instruct make it a strong candidate for:

• Customer support chatbots that handle Japanese and English queries.
• Language‑learning assistants offering explanations, translations, and practice dialogues.
• Code generation and review for developers working with C, C++, Java, Python, Rust, etc., thanks to its exposure to The Stack.
• Content creation such as blog drafting, summarization, and bilingual copy‑writing.
• Research prototyping in NLP, especially for cross‑lingual tasks and multilingual instruction following.

Training Details

The model was trained on a massive multilingual corpus totaling 2.1 trillion tokens. Pre‑training data includes:

• Japanese Wikipedia (2.6 B tokens) and Common Crawl (762.8 B tokens).
• English sources such as Wikipedia (4.7 B), Dolma/CC‑head (608.5 B), C4 (181.6 B), Reddit (83.1 B), and Gutenberg (5.5 B).
• Programming language data from The Stack (114.1 B tokens).
• Smaller Chinese and Korean Wikipedia corpora.

After the base pre‑training, the model underwent instruction tuning on the ichikara‑instruction‑004‑002 Japanese instruction dataset, which emphasizes clear task definitions and response formatting. Training was performed with torch>=2.3.0, transformers>=4.40.1, and flash‑attn>=2.5.8 to accelerate attention computation. The exact compute budget is not disclosed, but a 3.7 B model typically requires several thousand GPU‑hours on A100‑class hardware.

Fine‑tuning is straightforward via the Hugging Face AutoModelForCausalLM API, allowing users to adapt the model to domain‑specific instruction sets or to continue training on proprietary data.

Related Papers

While the README does not directly list academic publications, the model builds upon several well‑known research efforts:

• “LLaMA: Open and Efficient Foundation Language Models” – foundational architecture for decoder‑only LLMs.
• “The Stack: A Dataset of 3.2 TB of Code” – source of the code‑related pre‑training tokens.
• “Bilingual Language Modeling with Unigram Tokenizers” – methodology behind the custom Unigram byte‑fallback tokenizer.

These works collectively inform the model’s transformer design, tokenization strategy, and multilingual pre‑training pipeline.

Licensing Information

The model is released under the Apache 2.0 license, as indicated in the README. Apache 2.0 is a permissive open‑source license that allows commercial use, modification, distribution, and private use without requiring the source code to be disclosed. The only obligations are:

• Providing proper attribution to the original authors (llm‑jp and NII).
• Including a copy of the license in any redistributed binaries or source.
• Not using the trademark “llm‑jp” in a way that suggests endorsement.

Because the license is explicit, there are no “unknown” restrictions; you may integrate the model into SaaS products, internal tools, or commercial applications, provided you honor the attribution clause.