Technical Overview
Tarsier2‑Recap‑7b is a 7‑billion‑parameter video‑language model (VLM) that specializes in generating concise, high‑quality captions for short video clips. It is built on top of Qwen2‑VL‑7B‑Instruct and inherits the multimodal encoder‑decoder architecture that processes both visual frames and textual prompts. The model is further refined by distilling the video‑description expertise of the larger Tarsier2‑7b into a more lightweight 7‑B version.
Key Features & Capabilities
- Video Captioning: Generates fluent, fine‑grained descriptions of dynamic scenes, actions, and objects.
- Multimodal Prompting: Accepts a mixture of image frames, video URLs, and textual instructions.
- Zero‑Shot Q&A: Though trained only on caption data, it can answer multiple‑choice video questions when guided with appropriate prompts (e.g., TVBench).
- Efficient Inference: At 7 B parameters, it runs on a single high‑end GPU with 16 GB VRAM for modest batch sizes.
Architecture Highlights
- Base: Qwen2‑VL‑7B‑Instruct – a transformer‑based encoder that processes video frames (sampled at 1‑2 fps) and a decoder that generates text.
- Distillation Pipeline: Knowledge from Tarsier2‑7b (a 7 B model with strong video understanding) is transferred via supervised fine‑tuning on a curated caption dataset.
- Instruction Tuning: The model retains the instruction‑following behavior of Qwen2‑VL‑7B‑Instruct, allowing it to respond to “Describe the video” or “Summarize the clip in 2 sentences”.
Intended Use Cases
- Research on video‑language alignment and multimodal representation learning.
- Rapid prototyping of video captioning pipelines for media platforms.
- Educational tools that need short, accurate video summaries.
- Hobbyist projects exploring video‑question answering with minimal compute.
Benchmark Performance
Video‑language models are typically evaluated on two axes: caption quality and temporal reasoning. The most relevant benchmarks for Tarsier2‑Recap‑7b are:
- DREAM‑1K – a dense video description dataset that measures fine‑grained caption fidelity using F1‑score.
- TVBench – a multiple‑choice video question‑answering suite that tests temporal understanding across 10 sub‑tasks.
Results
- DREAM‑1K: Overall F1 = 40.7 % – only 1.3 % behind the full‑scale Tarsier2‑7b (42.0 %) and surpassing GPT‑4o’s 39.2 %.
- TVBench Overall: 54.0 % – comparable to Tarsier2‑7b’s 54.7 % despite being trained solely on caption data.
- Sub‑task highlights: Excellent “Action Antonym” (91.2 %) and “Scene Transition” (85.9 %) scores, indicating strong semantic and temporal discrimination.
These benchmarks matter because they capture both the model’s ability to describe visual content and to reason about events over time—core requirements for any practical VLM.
Hardware Requirements
VRAM & GPU
- Minimum: 16 GB VRAM (e.g., NVIDIA RTX 3080, RTX A6000) for single‑frame inference with batch size = 1.
- Recommended: 24 GB+ (e.g., RTX 4090, A100 40 GB) for batch processing of multiple clips or higher‑resolution frames.
CPU & RAM
- Modern multi‑core CPU (8 + cores) to handle data loading and preprocessing.
- At least 32 GB system RAM; 64 GB is advisable when working with long video sequences.
Storage
- Model checkpoint size ≈ 13 GB (safetensors format).
- Additional storage for the Tarsier2‑Recap‑585K fine‑tuning dataset (≈ 2 GB).
- Fast SSD (NVMe) recommended for low‑latency video frame extraction.
Performance Characteristics
- Inference latency: ~150 ms per 2‑second clip on a RTX 4090 (FP16).
- Throughput scales linearly with batch size up to GPU memory limits.
- Supports mixed‑precision (FP16/ BF16) for reduced memory footprint.
Use Cases
Primary Applications
- Automatic video caption generation for short clips (e.g., social‑media reels, educational snippets).
- Assistive technologies that provide audio descriptions for visually impaired users.
- Pre‑processing step for video‑question‑answering pipelines, where captions are fed into downstream QA models.
- Content moderation tools that need a textual summary of video content for policy checks.
Real‑World Examples
- Media platforms can auto‑generate subtitles for user‑uploaded videos, improving accessibility and SEO.
- Online learning portals can create concise video summaries to aid quick revision.
- Smart home devices can describe live camera feeds when queried by users.
Integration Possibilities
- Wrap the model in a REST API (FastAPI, Flask) and call it from web or mobile apps.
- Combine with speech‑to‑text modules to produce closed‑caption pipelines.
- Use as a feature extractor for downstream multimodal tasks (e.g., video retrieval, sentiment analysis).
Training Details
Methodology
- Base model: Qwen2‑VL‑7B‑Instruct (7 B parameters, vision‑language transformer).
- Fine‑tuning: Supervised training on the Tarsier2‑Recap‑585K dataset, which contains 585 K video‑caption pairs.
- Training schedule: 2 epochs with a learning rate of 2 × 10⁻⁵, using AdamW optimizer.
- Training date: December 2024.
Compute Requirements
- Estimated GPU hours: ~1,200 GPU‑hours on A100‑40 GB (mixed‑precision).
- Data pipeline: Frames sampled at 1–2 fps, stored as compressed JPEG/PNG to reduce I/O load.
Fine‑Tuning Capabilities
- Because the model retains the instruction‑following head of Qwen2‑VL, you can further fine‑tune it on domain‑specific caption data (e.g., medical video, sports highlights).
- Parameter‑efficient methods such as LoRA or QLoRA are compatible, allowing adaptation with as few as 10 M trainable parameters.
Licensing Information
The model card lists the license as unknown, but the underlying base model Qwen2‑VL‑7B‑Instruct is released under the Apache‑2.0 license. In practice, this means:
- Use: You may use, modify, and distribute the model for both research and commercial purposes, provided you comply with Apache‑2.0 terms.
- Attribution: You must retain the original copyright notice and include a copy of the Apache‑2.0 license in any distribution.
- Patent Grant: Apache‑2.0 includes an explicit patent license, reducing risk for commercial deployments.
- Potential Restrictions: If the final model incorporates additional data with a different license, you should verify that no conflicting terms exist. The “unknown” tag suggests checking the repository for any supplemental licensing notes.
For definitive legal guidance, consult the model’s Hugging Face model card and the official GitHub repository.