Tencent Hy3: Run a 295B Flagship MoE Model on a Single GPU with GGUF and llama.cpp

Tencent just made running a flagship-scale language model dramatically more accessible. Hy3 — the full production release of their Hunyuan 3 model series — is a 295B Mixture-of-Experts (MoE) model that activates only 21 billion parameters per token. And with the arrival of community 1-bit and 4-bit GGUF quantizations, you can now serve it on a single GPU using llama.cpp, complete with Multi-Token Prediction (MTP) for faster decoding.

This is one of the more significant open-weight releases of 2026: a model that competes with trillion-parameter flagships, ships under Apache 2.0, and is now genuinely reachable on consumer and prosumer hardware.

What Is Hy3? Architecture at a Glance

Hy3 uses a sparse Mixture-of-Experts architecture with 192 routed experts and top-8 routing — meaning only 8 experts fire per token, keeping inference compute far lower than the headline 295B parameter count suggests. Here's a quick summary of the key specs:

• Total parameters: 295B
• Active parameters per token: 21B (plus a 3.8B MTP/NextN speculative layer)
• Experts: 192 total, top-8 routing
• Context window: 256K tokens (community GGUF variants extend this to 1M)
• Architecture: hy_v3 (not yet merged into llama.cpp master — requires a PR branch build)
• License: Apache 2.0 — fully permissive for commercial use

For comparison, DeepSeek V4 Flash sits at 284B total parameters with roughly 13B active parameters per token. Hy3 is slightly larger in the active path but competitive in the same hardware tier. See our earlier breakdown of how DeepSeek V4-Pro permanently repriced enterprise AI for useful context on how MoE pricing and serving economics stack up.

What's New: 1-Bit and 4-Bit GGUF Quantizations

The community has moved quickly to make Hy3 accessible at the hardware level. The key quantization tiers now available as GGUF files are:

• IQ1_M (1-bit): ~83–92 GB — fits on a single high-memory GPU, though reasoning performance degrades at very low bit-depth
• Q2_K: ~100+ GB — good balance of quality and size for single-node serving
• Q4_K_M (4-bit): ~166–185 GB — the recommended quality sweet spot, typically needing 2–4 GPUs

The IQ1_M quant is the breakthrough for single-GPU deployment: it compresses the full 295B model into under 92 GB. That's tight, and you'll want to keep context lengths modest and consider quantizing the KV cache, but it works.

All community GGUF releases preserve the MTP/NextN layer at q8_0 precision — even on the lowest bit-depth tiers — so speculative decoding remains available and correct across all quantization levels.

Running Hy3 with llama.cpp

Important: The hy_v3 architecture is not yet in llama.cpp's master branch. You need to build from the PR branch until it merges. Here's the setup:

# Clone and build from the Hy3 port branch

Once built, serving is straightforward. For basic inference:

# Plain serve (no speculative decoding)

To enable MTP speculative decoding (up to ~37% faster decode on CUDA with an 88–90% acceptance rate):

# Serve with MTP self-speculative decoding

A few critical notes from the community:

• Always use --jinja: Hy3's default chat template uses .format() calls that llama.cpp's Jinja engine rejects. Several community repos include a fixed chat_template_llamacpp.jinja — pass it with --chat-template-file for tool calling.
• Memory-constrained? Offload MoE expert tensors to CPU with --n-cpu-moe N. This trades speed for capacity.
• First load warning: Large quantizations (100 GB+) can take several minutes to load — don't kill the process early.
• Tencent's recommended sampling: --temp 0.9 --top-p 1.0

Benchmark Highlights: Where Hy3 Excels

Tencent's internal benchmarks paint a strong picture for STEM, reasoning, and agentic tasks:

• GPQA Diamond: 90.4
• USAMO 2026: 72.0
• IMOAnswerBench: 90.0
• HLE (with tools): 53.2
• Blind expert test (270 experts, 312 comparisons): Hy3 scored 2.67/4, ahead of GLM-5.1 at 2.51 — with the largest edge in frontend development, CI/CD, and data/storage workflows

That said, community testing suggests some caveats: coding benchmarks show Hy3 at ~28% on DeepSWE vs GPT-5.4's 52%, and several practitioners note the model can be slower than DeepSeek V4 Flash. The tradeoff is reliability — real-world users report Hy3 stays on-task better in agentic workflows where DeepSeek V4 Flash can veer off course.

This is very much the same story playing out across the open-weight frontier — see our coverage of Moonshot AI's upcoming Kimi K3 and Z.ai's GLM-5.5 for more context on how Chinese labs are racing to push open-weight capability to new heights.

Serving at Scale: Multi-GPU with vLLM

For production deployments that need full-quality inference, Tencent recommends 8× H20 GPUs (or equivalent high-memory cards). The official vLLM recipe with MTP enabled:

vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy3

SGLang is also supported with similar MTP-enabled serving paths. For local multi-machine distribution, the same principles covered in our guide to Mesh LLM distributed inference apply here — pooling GPU memory across nodes can unlock larger quantization tiers without a monolithic server.

Free API on OpenRouter — For Now

Not ready to spin up local infrastructure? Tencent is offering a free API tier for Hy3 on OpenRouter for two weeks from launch (through approximately July 21, 2026), available at openrouter.ai/tencent/hy3. After that, pricing is expected to land in the DeepSeek V4 Flash tier — roughly $0.10–$0.30 per million input tokens.

Weights are available on Hugging Face, with a technical report at hy.tencent.com/research/hy3.

Should You Try Hy3?

If you're running agentic workflows and need a model that holds its reasoning across multi-step tasks, Hy3 is worth serious evaluation. The 1-bit GGUF tier makes single-GPU testing realistic for the first time. The Apache 2.0 license removes commercial friction entirely. And the free OpenRouter window means you can benchmark it against your real workloads before committing to infrastructure.

The caveats are real: llama.cpp support requires a PR-branch build for now, the IQ1_M quant sacrifices some reasoning depth, and peak coding scores trail GPT-5.4-class models. But as a locally deployable, commercially friendly, frontier-adjacent model for reasoning and agent tasks, Hy3 is one of the most compelling open-weight releases in months.

---

Related Reading

• DeepSeek V4-Pro Just Permanently Repriced Enterprise AI — Here's What That Actually Means
• Kimi K3 Launch Imminent: Moonshot AI's 2.5T-Parameter Model Set to Drop on July 15
• GLM-5.5 Could Arrive in August: Z.ai's 1 Trillion Parameter Beast Takes Aim at the West
• Mesh LLM: Run Any Model Across Multiple Machines With One OpenAI-Compatible API