colibrì: How a Single C File Runs a 744B-Parameter MoE Model on 25 GB of RAM

Running massive AI models locally usually requires expensive GPUs. However, colibrì: How a Single C File Runs a 744B-Parameter MoE Model on 25 GB of RAM proves otherwise. This dependency-free C engine runs GLM-5.2 on consumer hardware with minimal memory. It achieves this by streaming experts directly from an NVMe disk.

The model has 744 billion total parameters but activates only 40B per token. Furthermore, just 11B parameters change between tokens. colibrì exploits this gap efficiently. It uses roughly 2,400 lines of C code without Python or CUDA dependencies. The engine loads a dense core into RAM while streaming quantized experts layer by layer. Consequently, it boots in about 32 seconds on a standard laptop.

How colibrì Splits Storage Tiers for Efficiency

GLM-5.2 is a Mixture-of-Experts transformer with 75 layers. colibrì divides the model into two distinct storage tiers to manage resources effectively.

• Dense part: Attention and embeddings stay resident in RAM. This 17B parameter slice uses ~9.9 GB.
• Routed experts: The system streams 21,504 individual experts from disk. These total roughly 370 GB at int4 quantization.

This approach mirrors virtual memory paging applied to transformer weights. Therefore, most of the model never occupies RAM simultaneously. Only the specific slice needed for a given token loads into memory.

Key Technical Features Beyond Proof-of-Concept

This project faithfully reproduces the GLM-5.2 architecture. Validation confirms token-exact output against Hugging Face Transformers oracles. Several advanced features enable this performance.

Optimized Attention and Routing

MLA attention compresses the KV-cache significantly. It uses 576 floats per token instead of 32,768. Additionally, a DeepSeek-V3-style sigmoid router manages expert selection efficiently. Native MTP speculative decoding drafts tokens using the model’s own prediction head. At int8 quantization, acceptance rates reach 39-59%. Grammar-forced drafts further boost efficiency during constrained generation like JSON.

Persistence and Prefetching

KV-cache persistence allows conversations to survive restarts. The compressed cache appends to disk after every turn. Moreover, experimental router-lookahead prefetch predicts the next layer’s routing. Tests show 71.6% predictability from current states. Hand-written AVX2 intrinsics power the quantization kernels. Decisions rely on measurement rather than assumption.

Honest Performance Metrics for colibrì

Transparency matters more than marketing numbers. On reference hardware with 25 GB RAM, cold decode costs ~11 GB of disk reads per token. Throughput ranges from 0.05 to 0.1 tokens per second. The project openly admits this speed is slow due to VHDX layer limitations.

However, accuracy remains correct on affordable hardware. Evaluate colibrì based on accuracy-per-dollar rather than raw speed. It suits batch processing or offline research perfectly. First, run iobench to measure your actual random-read throughput. Then, set your RAM budget as high as safely possible.

Community Benchmark Insights

User benchmarks reveal clear patterns regarding hardware bottlenecks. Machines with limited RAM cap expert cache slots regardless of disk speed. Conversely, faster NVMe drives improve throughput until CPU kernels become the limit. For example, upgrading to PCIe 5.0 storage nearly tripled token generation speed.

Speculative Decoding and Learning Cache Benefits

MTP speculative decoding uses the trained draft head for high acceptance rates. Yet, cold caches may load extra experts during verification. This can temporarily reduce speed until warming occurs. An adaptive guard manages this trade-off automatically.

Additionally, a learning cache improves performance over time. It logs expert usage and pins hot experts in spare RAM. Benchmarks show hit rates climbing from 28% to 66% through repeated use. No code changes are required for these gains.

Getting Started with Installation and Serving

Setup involves downloading shards and converting them to int4 containers. Alternatively, pre-converted weights exist on Hugging Face. Always run coli doctor first to validate readiness. This prevents out-of-memory errors before loading the massive model.

The included OpenAI-compatible server exposes standard chat endpoints. It supports SSE streaming and reasoning modes. Note that concurrent requests queue through a bounded FIFO system. Isolated KV-cache slots allow up to 16 independent contexts. Finally, remember that this tool excels at offline workloads rather than interactive chatbot replacement.