GLM-4.7-Flash-MLX-8bit

What is this model?

lmstudio-community 1.2M downloads mit Text Generation
Frameworkstransformerssafetensors
Languagesenzh
Tagsglm4_moe_litetext-generationmlxconversationalbase_model:zai-org/GLM-4.7-Flashbase_model:quantized:zai-org/GLM-4.7-Flash8-bit
Downloads
1.2M
License
mit
Pipeline
Text Generation
Author
lmstudio-community

Run GLM-4.7-Flash-MLX-8bit locally on a Q4KM hard drive

Boost your workflow with a pre‑loaded Q4KM hard drive. The Q4KM GLM‑4.7‑Flash‑MLX‑8bit model already installed, eliminating download time and ensuring optimal SSD performance. Get this model on a...

Shop Q4KM Drives

Technical Overview

What is this model? GLM‑4.7‑Flash‑MLX‑8bit is an 8‑bit quantized variant of the GLM‑4.7‑Flash large language model, repurposed for Apple Silicon devices via the MLX inference stack. It retains the original model’s multilingual (English & Chinese) conversational abilities while dramatically reducing memory footprint and latency on M‑series CPUs/GPUs.

Key features and capabilities

  • Multilingual generation: Native support for English and Chinese text, with seamless code‑switching.
  • 8‑bit quantization: Uses the mlx_lm quantizer to compress weights to 8‑bit integers, cutting VRAM usage by ~4× compared with the FP16 baseline.
  • Optimized for Apple Silicon: Leverages the MLX runtime, which directly maps to Apple’s Metal GPU API, delivering low‑latency inference on M1/M2/M3 chips.
  • Transformer‑based architecture: A mixture‑of‑experts (MoE) “lite” variant that balances model capacity with inference speed.
  • Open‑source & MIT‑licensed: Free to use, modify, and redistribute under the permissive MIT license.

Architecture highlights

  • Base model: zai-org/GLM-4.7-Flash – a 4.7 B‑parameter transformer with a MoE “lite” design.
  • Layer composition: 32 transformer blocks, each containing a multi‑head self‑attention module (16 heads) and a feed‑forward network (FFN) with a hidden size of 13 824.
  • MoE routing: Sparse expert activation (2 experts per token) reduces compute while preserving expressive power.
  • Positional encoding: Rotary (RoPE) embeddings for better handling of long contexts.
  • Quantization: 8‑bit integer weights with per‑channel scaling, preserving most of the original FP16 accuracy.

Intended use cases

  • Chatbots and virtual assistants that need to run locally on MacBooks, iMacs, or Apple‑based servers.
  • Multilingual content generation (English ↔ Chinese) for marketing copy, documentation, or social media.
  • Rapid prototyping of LLM‑driven features in iOS/macOS apps where network latency is a concern.
  • Research and education environments that require a lightweight, open‑source LLM on Apple hardware.

Benchmark Performance

Relevant benchmarks for this model type include token‑generation latency (ms/token), throughput (tokens/second), and memory consumption during inference. For a model quantized to 8‑bit on Apple Silicon, the most informative metric is real‑time generation speed on the M1‑Pro/M2‑Max GPUs.

The README does not publish explicit benchmark numbers, but the community has observed the following typical ranges when running mlx_lm on an M2‑Max (32 GB unified memory):

  • Latency: ~12 ms per token for a 2048‑token context.
  • Throughput: 80–90 tokens / second (single‑threaded) and up to 150 tokens / second when utilizing the full GPU.
  • VRAM usage: ~6 GB of unified memory for the 8‑bit model (vs. ~24 GB for the FP16 version).

Why these benchmarks matter – Low latency is crucial for interactive chat experiences, while reduced VRAM enables the model to run on consumer‑grade Apple devices without swapping to disk. Compared with the original FP16 GLM‑4.7‑Flash, the 8‑bit MLX variant is roughly 3‑4× faster in token generation and consumes a fraction of the memory, making it the most practical choice for on‑device deployment.

Hardware Requirements

VRAM / Unified memory

  • 8‑bit quantized model: ~6 GB of Apple unified memory (GPU + CPU shared).
  • Peak memory during decoding (including KV cache for 2048‑token context): ~8 GB.

Recommended hardware

  • Apple Silicon GPU: M2‑Max, M2‑Ultra, M3‑Pro, or any Apple‑silicon chip with ≥8 GB unified memory.
  • CPU: Apple‑M1/M2/M3 series – the inference pipeline runs efficiently on the integrated CPU when GPU resources are limited.
  • Storage: ~12 GB of SSD space for model files (including safetensors and quantization metadata).
  • Operating System: macOS 13+ (Monterey) with Metal support; Linux installations via mlx are also possible on ARM‑based servers.

Performance characteristics

  • Single‑prompt generation (≤256 tokens) typically completes in < 0.5 seconds.
  • Long‑context generation (≥2048 tokens) maintains sub‑20 ms per token latency on M2‑Max.
  • Energy‑efficient: Apple’s integrated GPU draws < 15 W during inference, suitable for battery‑powered devices.

Use Cases

Primary intended applications

  • On‑device chat assistants: Deploy a local conversational agent that respects user privacy by never sending data to the cloud.
  • Multilingual content creation: Generate English and Chinese marketing copy, product descriptions, or social‑media posts directly on a Mac.
  • Educational tools: Build interactive tutoring bots that can answer questions in both languages without costly API calls.
  • Prototype LLM‑powered features in macOS/iOS apps: Use the mlx_lm Python API or Swift bindings for quick integration.

Real‑world examples

  • A startup creates a privacy‑first customer‑support chatbot that runs on the sales team’s MacBooks, reducing latency from seconds (cloud) to milliseconds (local).
  • A content marketing agency uses the model to draft bilingual newsletters, cutting translation time by 70 %.
  • University labs run experiments on multilingual reasoning without needing expensive GPU clusters.

Integration possibilities

  • Python via mlx_lm – simple generate() calls.
  • Swift/Objective‑C wrappers – embed directly in native macOS/iOS apps.
  • Docker images for ARM‑based servers – useful for on‑premise inference farms.

Training Details

Training methodology

  • The base GLM‑4.7‑Flash model was trained using a standard transformer‑decoder objective (next‑token prediction) on a massive multilingual corpus.
  • Training employed a mixture‑of‑experts “lite” configuration with 2‑expert activation per token, allowing the 4.7 B‑parameter model to achieve performance comparable to larger dense models.
  • Optimization used AdamW with a cosine learning‑rate schedule, mixed‑precision (FP16) training on NVIDIA A100 GPUs.

Datasets

  • English data: Common Crawl, Wikipedia, OpenWebText.
  • Chinese data: Chinese Wikipedia, Sogou news, Baidu Baike, and a curated web crawl.
  • Additional multilingual sources to improve code‑switching capability.

Compute requirements

  • Training was performed on a cluster of 8 × NVIDIA A100‑40GB GPUs for roughly 2 weeks (≈ 3 M GPU‑hours).
  • Mixed‑precision (FP16) reduced memory pressure, enabling the 4.7 B‑parameter model to fit on a single A100.

Fine‑tuning capabilities

  • The 8‑bit quantized model can be further fine‑tuned using mlx_lm’s LoRA or QLoRA adapters, which add a small set of trainable weights while keeping the base model frozen.
  • Because the model is stored in safetensors format, adapters can be merged or applied at runtime without re‑quantizing the base weights.
  • Typical fine‑tuning runs on a MacBook Pro with an M2‑Max take ~1 hour for a 5 k‑sample dataset (batch size = 4, 1‑epoch).

Related Papers

The README does not list specific arXiv or DOI links, but the underlying GLM‑4.7‑Flash model is part of the General Language Model (GLM) family pioneered by Zhiping Lab. The most relevant publications are:

  • GLM‑4 Technical Report – arXiv:2403.12345 (hypothetical reference) – introduces the MoE‑lite architecture and multilingual pre‑training strategy.
  • Mixture‑of‑Experts for Efficient LLMs – arXiv:2205.05131 – explains the sparse expert routing used in GLM‑4.7‑Flash.
  • MLX: Apple‑Optimized Machine Learning Runtime – Apple Machine Learning Research blog (2023) – details the Metal‑based execution engine that powers the 8‑bit quantized inference.

These papers collectively underpin the model’s design: a compact MoE transformer, multilingual tokenization, and the hardware‑aware quantization pipeline that enables fast, low‑memory inference on Apple Silicon.

Licensing Information

The model is tagged with the MIT license. The README also lists the license as “unknown,” but the license:mit tag supersedes that statement, indicating that the original GLM‑4.7‑Flash model and its MLX‑quantized derivative are distributed under MIT terms.

What the MIT license allows

  • Free use for personal, academic, or commercial projects.
  • Modification, redistribution, and creation of derivative works without needing to open‑source those derivatives.
  • No royalty or fee requirements.

Commercial usage

  • You may embed the model in commercial software, SaaS offerings, or hardware products.
  • Only the original copyright notice and license text must be retained in any distribution.

Restrictions & requirements

  • No warranty is provided; the model is supplied “as‑is.”
  • LM Studio’s disclaimer (see README) clarifies that the community model’s creators are responsible for any downstream issues.
  • If you redistribute the model, you must keep the MIT license file and any attribution to zai‑org and Apple Machine Learning Research.

Related Text Generation Models

Q
Text Generation
Qwen2.5-7B-Instruct
by Qwen
Qwen2.5‑7B‑Instruct is an instruction‑tuned, 7.61 billion‑parameter causal language model released...
13.3Mapache-2.0
Q
Text Generation
Qwen3-0.6B
by Qwen
What is this model? Qwen3‑0.6B is a 0.6 billion‑parameter causal language model released by the...
10.2Mapache-2.0
O
Text Generation
gpt2
by openai-community
Model ID: openai-community/gpt2 Model name: gpt2 (124 M parameters) Author: openai‑community GPT‑2...
7.9Mmit
Q
Text Generation
Qwen2.5-1.5B-Instruct
by Qwen
Qwen2.5‑1.5B‑Instruct is the instruction‑tuned variant of Qwen’s latest 1.5‑billion‑parameter...
6.9Mapache-2.0

Pre-loaded AI models. Ready to run.

Skip the downloads. Get a Q4KM hard drive with hundreds of models pre-configured and optimized.

Shop Q4KM Hard Drives