Gemma 4's Multi-Token Prediction: How Google Made Its Smaller Models Inference Speed Monsters

Quick Answer

Google’s Gemma 4 uses a technique called multi-token prediction to speed up inference. Instead of generating one token, checking it, then moving to the next, a smaller drafter model proposes multiple tokens at once and the main model verifies them in parallel. The result: 2-3x faster inference on the same hardware.

The practical upside: Gemma 4 26B and 31B are free on OpenRouter’s free tier, with standard pricing of $0.06-0.13 per million input tokens. For context, Claude 3.5 Sonnet charges $3.00 per million input tokens. The capability gap is real for complex reasoning, but for coding assistance, summarization, and standard AI tasks, Gemma 4 is fast becoming a credible alternative.

---

What Multi-Token Prediction Actually Is

Standard language model decoding is sequential. The model predicts token 1, then uses token 1 to predict token 2, then token 2 to predict token 3, and so on. Every token wait for the one before it.

Multi-token prediction changes this. A secondary model — the drafter — proposes a small sequence of tokens at once. The main model then verifies all of them in a single forward pass. If the drafter was right, you just saved multiple forward passes. If it was wrong on token 3, the main model corrects it and continues.

The key insight is that verification is cheaper than generation. A smaller model can propose tokens cheaply, and a larger model can verify them quickly in parallel. The net effect is fewer total compute steps for the same output length.

Google’s implementation in Gemma 4 uses what they call “drafters” — lightweight predictors that run alongside the main model. The drafter doesn’t generate text on its own; it previews what the main model might say next, and the main model either accepts or rejects each proposed token.

The technique has been explored in academic literature for years (see the Medusa and EAGLE papers from 2023-2024), but Google’s implementation in a production frontier-adjacent model is what brought it to mainstream attention.

---

What the Speedup Looks Like in Practice

In Google’s published benchmarks, multi-token prediction delivers 2-3x throughput improvement over standard autoregressive decoding at equal quality. The gains are most pronounced on longer outputs where the pipeline can fully warm up.

For a concrete example: a coding task that takes 30 seconds with standard decoding might complete in 10-12 seconds with multi-token prediction enabled. On workloads with sustained token generation, the improvement compounds.

The Hacker News discussion around Gemma 4’s release surfaced real-world reports that line up with this. One developer noted that Gemini CLI — which uses related techniques — had improved dramatically in recent months, going from frustrating loops to reliably solving complex build problems in minutes. Another compared it favorably to Claude on technical tasks after being stuck on a problem with the latter.

These are anecdotal, but the pattern matches what you’d expect from the architecture: tasks with clear structure and moderate complexity benefit most. Ambiguous reasoning tasks still benefit from frontier models.

---

OpenRouter Pricing — Gemma 4 vs. the Field

Gemma 4 costs on OpenRouter right now, compared to the models it competes with:

Model	Context	Input Cost	Output Cost	Free Tier
Gemma 4 26B	262K	$0.06/M	$0.33/M	Yes
Gemma 4 31B	262K	$0.13/M	$0.38/M	Yes
Claude 3.5 Sonnet	200K	$3.00/M	$15.00/M	No
GPT-4o	128K	$2.50/M	$10.00/M	No
DeepSeek V3	64K	$0.01/M	$0.03/M	No
Qwen 3.5 35B	262K	$0.15/M	$1.00/M	No

Prices from OpenRouter (May 2026). Verify before making infrastructure decisions.

Gemma 4 sits between the ultra-cheap DeepSeek tier and the premium frontier models. For teams that need more capability than DeepSeek offers but want to avoid Claude/GPT pricing, it’s a legitimate middle ground — especially given the 262K context window.

---

What You Can Actually Build With It

The combination of multi-token prediction, long context, and low cost opens up some specific use cases where Gemma 4 makes sense.

Code generation and review. With a 262K context, you can feed an entire medium-sized codebase into Gemma 4 and ask it to generate or review code. The multi-token prediction speedup matters here — longer code outputs are where the technique pays off most.

Document processing at scale. Summarizing, extracting, or classifying long documents. The context window means you don’t need chunking strategies, and the throughput is high enough for production workloads.

Developer tooling. The HN discussion mentioned Gemini CLI competing with Claude for developer tasks. Gemma 4 occupies similar territory — fast enough for interactive use, capable enough for real work.

High-volume AI tasks. If you’re running thousands of daily tasks that don’t require frontier reasoning — classification, extraction, straightforward Q&A — Gemma 4’s free tier covers substantial volume.

---

Where It Falls Short

Gemma 4 is not a Claude replacement for complex reasoning. The multi-token prediction speedup comes from architectural optimization, not from raw capability improvements. If a task requires multi-step reasoning about an ambiguous problem, frontier models still outperform.

The practical gap shows up in:

• Complex debugging where the solution path isn’t obvious
• Tasks requiring deep domain expertise and nuanced judgment
• Problems where you need the model to challenge your assumptions

For these, you’re still paying for Sonnet or GPT-4o.

There’s also the rate limit consideration on the free tier. OpenRouter’s free tier works for development, but production traffic needs the standard tier. At $0.06-0.13/M input, it’s still far cheaper than alternatives — but it’s not free at scale.

---

Should You Switch?

The calculus is straightforward.

If you’re paying for Claude or GPT-4o on tasks that don’t require frontier reasoning — and you probably are, most teams are — Gemma 4 is worth testing. Spin up the free tier, run your actual workloads against it, and compare results.

If Gemma 4 passes your quality bar for 60% of your tasks, you’ve just cut your API bill by 90%+ on that portion. The remaining 40% (complex reasoning, nuanced judgment, ambiguous problems) stays with the frontier model.

The bottleneck is rarely the model. It’s usually the assumption that you need the most expensive option for every task. Gemma 4 multi-token prediction is Google’s argument that speed and cost can improve without sacrificing capability — and for a growing class of production AI tasks, they’re right.

Related Reading:

• OpenRouter Pricing Guide 2026 — full model aggregation breakdown
• How Much Does Claude 3.5 Sonnet Cost? — Sonnet vs Gemma pricing
• Small Language Models: How to Stop Overpaying — when smaller models make sense
• DeepSeek V3 Cost Analysis — another cheap capable model

Community & Sources:

• HN: Accelerating Gemma 4
• Google AI Blog: Multi-Token Prediction

---

Pricing data from OpenRouter (May 2026). Gemma 4 benchmark data from Google’s official publications and community reports. Results vary by workload — test with your actual tasks before committing to any model for production use.