The accepted wisdom in large language model deployment has always been simple: bigger models need bigger hardware. Running a 70-billion-parameter model requires roughly 140GB of GPU memory in full precision — the equivalent of two A100 GPUs running in tandem. AirLLM, an open-source Python library by developer Gavin Li, breaks that assumption entirely. It runs 70B models on a single 4GB consumer GPU — without quantization, distillation, or pruning — by rethinking how models interact with memory at inference time.
The project, hosted at github.com/lyogavin/airllm, has quietly grown into one of the most practically significant tools in the open-source AI stack. With support for every major model family — Llama 2/3/4, Qwen, DeepSeek V2/V3/R1, Mistral, Mixtral, Gemma, Phi, ChatGLM, Baichuan, InternLM, Yi — and a one-line API that mirrors the standard HuggingFace interface, it's become a serious option for researchers, developers, and hobbyists priced out of multi-GPU setups.
The Core Insight: You Don't Need the Whole Model at Once
A transformer model like Llama 3 70B is composed of roughly 80 stacked layers. Traditional inference loads all of them into GPU memory simultaneously, which is why the VRAM requirement scales linearly with parameter count. AirLLM's breakthrough is deceptively simple: only one layer ever lives on the GPU at a time.
During a forward pass, AirLLM loads a single layer from disk, computes the activations, offloads that layer back to storage, then prefetches the next. The KV cache — the memory-intensive component that grows with input length — stays compact. For a 70B model with 100 tokens of input, that cache works out to roughly 30MB, well within any modern GPU's headroom. The result is that peak VRAM usage stays under 4GB regardless of the model's total size.
The technical sequence looks like this:
- Layer N loads from disk (or CPU RAM) onto the GPU
- Activations are computed for the current sequence
- Layer N is offloaded; Layer N+1 begins prefetching in parallel
- Repeat across all 80 layers to complete the forward pass
Prefetching — introduced in v2.5 — overlaps disk I/O with GPU compute, delivering roughly a 10% speed improvement by eliminating the idle gap between layer transitions.
Flash Attention and the Memory Math
Layer sharding alone wouldn't be enough without parallel work on the attention mechanism itself. Standard self-attention requires O(n²) memory relative to sequence length — memory that grows quadratically as inputs get longer. AirLLM integrates Flash Attention, which rewrites the attention computation to avoid materializing the full intermediate attention matrix. This keeps the attention footprint manageable even as context length grows, and is now standard across virtually every production LLM.
Together, layer decomposition plus Flash Attention compress peak memory from 140GB+ to under 4GB — a 97% reduction that preserves the full-precision model with no accuracy loss.
Compression: When Speed Matters More Than Purity
Running a full-precision 70B model on a 4GB GPU is genuinely possible with AirLLM, but the disk I/O bottleneck means inference is slow. For workloads where throughput matters, AirLLM offers optional weight-only quantization at 4-bit or 8-bit precision — delivering around 3× runtime speedup with minimal accuracy impact.
The key distinction here is that AirLLM only quantizes weights, not activations. Quantizing activations (as standard quantization schemes do to maximize hardware throughput) introduces error from input outliers. Since AirLLM's bottleneck is disk bandwidth rather than compute, weight-only quantization is sufficient — and considerably safer for accuracy.
pip install airllm
from airllm import AutoModel
model = AutoModel.from_pretrained(
"meta-llama/Meta-Llama-3-70B-Instruct",
compression='4bit' # or '8bit', or omit for full precision
)
The same AutoModel.from_pretrained() call works for every supported model family. AirLLM auto-detects the model type from the HuggingFace repo ID — no manual class selection required since v2.6.
What It Actually Supports
AirLLM's model compatibility has grown substantially since its initial Llama 2 launch in late 2023. As of the latest releases, it works out-of-the-box with:
- Llama 2, 3, 3.1, 3.3, and 4
- Qwen 1, 2, 2.5, and 3 (including MoE and FP8 variants)
- DeepSeek V2, V3, and R1
- Mistral and Mixtral
- Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi
The 405B Llama 3.1 model — a behemoth that normally demands cluster-level infrastructure — runs on 8GB VRAM with AirLLM's layer sharding active. Even DeepSeek R1 at 671B parameters is reachable on hobbyist hardware, with VRAM requirements determined by individual layer size rather than total model size.
CPU inference, added in v2.10.1, extends this further: machines with no discrete GPU at all can now run inference, trading speed for complete hardware flexibility.
macOS Support
Since v2.8.2, AirLLM has supported Apple Silicon Macs, enabling 70B inference on M-series machines. For developers on MacBooks who want to run research-grade open models locally without cloud costs, this makes AirLLM one of the most practical options available — especially for models that don't yet have optimized Metal backends elsewhere.
The Trade-Off: This Isn't a vLLM Replacement
AirLLM's approach trades inference speed for memory efficiency. Layer-by-layer loading introduces latency that multi-GPU batch inference doesn't have. For production serving with high concurrency and throughput requirements, frameworks like vLLM remain the appropriate choice.
Where AirLLM excels is in a different set of use cases:
- Research and prototyping on hardware you already own
- Edge deployment in offline or bandwidth-constrained environments
- Education — universities can now offer hands-on 70B model access in standard computer labs
- Privacy-sensitive applications where cloud inference is off the table
- Cost-constrained development where renting A100 clusters isn't viable
This positions AirLLM closer to the spirit of llama.cpp than to vLLM — a tool for making powerful models accessible, not for maximizing throughput at scale. The comparison with running massive quantized models via GGUF and llama.cpp is instructive: both sacrifice some speed to democratize access, but AirLLM does so without touching model weights at all in its default mode.
It's worth reading alongside recent developments in on-device and low-resource inference more broadly — for instance, Bonsai 27B's approach to running frontier-class models on smartphones and how Tencent's Hy3 uses GGUF quantization to fit a 295B MoE model on a single consumer GPU. Together these projects sketch a picture of what inference democratisation actually looks like in practice.
Getting Started
Installation is a single pip command:
pip install airllmA basic inference run for Llama 3 70B looks like this:
from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("meta-llama/Meta-Llama-3-70B-Instruct")
input_text = ['What is the capital of France?']
input_tokens = model.tokenizer(
input_text,
return_tensors="pt",
return_attention_mask=False,
truncation=True,
max_length=MAX_LENGTH,
padding=False
)
generation_output = model.generate(
input_tokens['input_ids'].cuda(),
max_new_tokens=20,
use_cache=True,
return_dict_in_generate=True
)
output = model.tokenizer.decode(generation_output.sequences[0])
print(output)
On first run, AirLLM splits the model checkpoint into per-layer shards (typically 80–100 files for a 70B model) and saves them to disk. Subsequent runs load directly from these shards — the one-time preprocessing cost is front-loaded, not repeated.
What's Next
AirLLM's trajectory suggests its scope will continue to expand. The addition of CPU inference, Apple Silicon support, and same-day compatibility with newly released models (the project claims most new HuggingFace models work immediately) reflects an active development posture. The patent-pending layer decomposition engine suggests Gavin Li has longer-term commercial or licensing ambitions beyond the current MIT open-source release.
The more interesting question is what AirLLM's existence implies for the industry. If 70B models genuinely run on hardware sold at consumer electronics retailers, the assumption that frontier-class open models require enterprise infrastructure starts to look fragile. That has real implications for how AI developers — particularly those navigating the real costs of AI in production — think about their infrastructure footprint. It also changes the calculus for anyone evaluating open versus closed models from a security and auditability standpoint.
The GitHub repository is at github.com/lyogavin/airllm. It's MIT-licensed, actively maintained, and — unlike most tools in this space — genuinely does what it says on the tin.
