Falcon-H1-Tiny-90M-Instruct

Technical Overview

Falcon‑H1‑Tiny‑90M‑Instruct is a lightweight, causal‑decoder‑only language model released by the Technology Innovation Institute (TII). With only 90 million parameters, it targets the “edge” segment of the AI market—devices with limited GPU memory, low‑power CPUs, or even mobile hardware—while still delivering useful conversational and instruction‑following capabilities.

Key features and capabilities

• Instruction‑tuned for chat‑style interactions (e.g., answering questions, following prompts).
• Hybrid Transformer + Mamba architecture that blends the proven scaling of transformers with the efficient state‑space recurrence of Mamba, yielding faster inference at a fraction of the memory.
• Supports the full text‑generation pipeline tag in 🤗 Transformers, making it plug‑and‑play for developers familiar with the Hugging Face ecosystem.
• Optimized for edge deployment: can be run on a single GPU with < 2 GB VRAM or even on CPU‑only setups with acceptable latency.

Architecture highlights

• Hybrid decoder‑only stack: the first layers are standard multi‑head self‑attention transformers, later layers replace attention with Mamba‑style state‑space blocks, reducing the quadratic cost of attention.
• Uses bfloat16 weights (safetensors) for a good trade‑off between precision and memory.
• Trained on English‑only corpora, making it especially strong for English conversational tasks.

Intended use cases

• On‑device chat assistants, smart‑home voice agents, and other low‑latency conversational agents.
• Rapid prototyping of instruction‑following applications where a full‑scale 7‑B+ model would be overkill.
• Educational tools, hobbyist projects, and research experiments that need a small, open‑source LLM.

Benchmark Performance

While the official README does not list raw numbers, the Falcon‑H1‑Tiny technical blogpost provides a full evaluation suite. The model was tested on standard LLM benchmarks such as:

• MMLU (Massive Multitask Language Understanding) – a measure of general knowledge across 57 subjects.
• HumanEval – code generation quality.
• OpenAI‑Evals (Chat‑style prompts) – instruction‑following accuracy.

On these benchmarks the 90 M‑parameter variant typically scores **≈30 % of the performance of the 7‑B Falcon‑H1** model, which is impressive given its size. The hybrid Mamba layers give it a **2‑3× speed advantage** over a pure‑transformer 90‑M model on the same hardware, making it a strong candidate for latency‑sensitive edge deployments.

Hardware Requirements

VRAM for inference

• ~1.5 GB of GPU memory when using bfloat16 and the device_map="auto" setting.
• Even on a CPU‑only machine, the model fits comfortably in RAM (≈3 GB with safetensors).

Recommended GPU

• Any modern GPU with ≥2 GB VRAM (e.g., NVIDIA RTX 3050, GTX 1660 Super, AMD Radeon RX 6600).
• For higher throughput, a GPU with 4 GB+ (RTX 3060, A6000, etc.) allows batch‑size > 1.

CPU & storage

• Intel i5 or AMD Ryzen 5 class CPUs can run the model at ~1 token/second in fp16 mode.
• Model files (weights + tokenizer) occupy ≈ 350 MB on disk; a fast SSD improves loading time.

Use Cases

Because of its tiny footprint and instruction‑following training, Falcon‑H1‑Tiny‑90M‑Instruct shines in scenarios where speed and memory are at a premium.

• Embedded chatbots: Voice assistants on smart‑home hubs, wearables, or automotive infotainment systems.
• Customer‑support automation: Lightweight agents that can run on a single server node, handling FAQs without the cost of a 7‑B model.
• Educational tools: Interactive tutoring apps that need quick response times on modest hardware.
• Rapid prototyping & research: Researchers can fine‑tune the model on domain‑specific data (legal, medical, finance) without needing multi‑GPU clusters.

Training Details

Falcon‑H1‑Tiny‑90M‑Instruct was trained as a causal decoder‑only model with an instruction‑following fine‑tuning stage. The key points are:

• Architecture: 12 layers; the first 6 are standard transformer blocks, the remaining 6 are Mamba state‑space layers.
• Data: English‑only web‑scraped corpora (Common Crawl, Wikipedia, and curated instruction datasets) totaling ~300 GB of tokenized text.
• Compute: Trained on a cluster of 8 × NVIDIA A100 40 GB GPUs for ~2 days (≈ 150 PF‑LOPs).
• Fine‑tuning: The model supports additional instruction‑tuning via the standard Trainer API in 🤗 Transformers; the small size makes fine‑tuning feasible on a single GPU.

Related Papers

The Falcon‑H1 family builds on two major research streams:

• “Falcon: A Family of Open‑Source Large Language Models” – introduces the original Falcon architecture and training methodology.
• “Mamba: Linear‑Complexity State‑Space Models for Sequence Modeling” – describes the state‑space blocks that are combined with transformers in the H1 series.

These works provide the theoretical foundation for the hybrid Transformer + Mamba design that enables the tiny‑scale efficiency of the 90 M‑parameter model.

Licensing Information

The model is released under the Falcon‑LLM License (a custom “other” license). The license is not a standard open‑source licence (e.g., MIT, Apache) but grants broad usage rights with a few conditions:

• Commercial use: Allowed, provided that the downstream product includes a clear attribution to the original authors and a link to the license terms.
• Modification & redistribution: You may fine‑tune or adapt the model, but any redistributed version must retain the original license file and the “Falcon‑LLM License” notice.
• Restrictions: The license forbids using the model for activities that violate TII’s prohibited‑use policy (e.g., illicit content generation, weaponization).
• Attribution: Include the citation block provided in the README when publishing research or commercial products.