DiffusionGemma 26B: Google's Text-Diffusion Model Hits 1100 Tokens/Sec — What Builders Actually Need to Know

On June 10, 2026, Google DeepMind released DiffusionGemma 26B-A4B-it — a model that replaces the standard one-token-at-a-time autoregressive inference loop with a text diffusion approach that denoises blocks of up to 256 tokens in parallel per forward pass. The result: 1100+ tokens per second on an H100 in FP8 at low batch sizes, which Google describes as up to 4x faster token output than Gemma 4 on dedicated GPUs (Google notes this speedup applies to local, low-concurrency inference, with diminishing returns in high-QPS cloud serving).

It’s open-weight, Apache 2.0, and fits in roughly 18GB of VRAM when quantized to NVFP4, with day-zero support in vLLM, Hugging Face Transformers, MLX, and Unsloth (NVIDIA NeMo also has day-zero support; llama.cpp support is coming later).

The trade-off is real: DiffusionGemma scores substantially lower than Gemma 4 on reasoning, math, and coding (see benchmarks below). Google’s own developer guide calls DiffusionGemma “this experimental model,” and its announcement post states: “For applications that demand maximum quality, we recommend deploying standard Gemma 4.” For builders, the question is whether the speed profile makes sense for your specific use case — and in several cases, it might.

What Text Diffusion Actually Means (and Why It’s Fast)

Standard autoregressive language models generate one token at a time. Each token depends on the previous tokens, so the process is inherently sequential. You can batch multiple requests, but you can’t parallelize the generation of a single response.

Text diffusion works differently. The model starts with a “canvas” of up to 256 masked tokens and iteratively denoises them — predicting the most likely tokens, committing the high-confidence ones, and re-running the diffusion pass on the remainder. The sampler is capped at 48 maximum denoising steps, but adaptive stopping (triggered once average per-canvas entropy drops below threshold) means a response typically finishes in about 12–16 steps in practice rather than the full 48.

The architecture uses an encoder-decoder design: an autoregressive encoder processes and caches the prompt context (causal attention, prefill), then a bidirectional-attention decoder runs the diffusion passes over the generation canvas, reading — not writing — the cached context via cross-attention. The bidirectional attention in the decoder is what allows parallel token prediction — unlike autoregressive models, the decoder can “see” the entire canvas position at once, not just positions to the left.

This is the same principle that made diffusion models dominant in image generation (where every pixel can be denoised in parallel). DiffusionGemma applies it to discrete text tokens.

Model Specs

All specs below are from Google’s official model card for DiffusionGemma 26B-A4B-it.

The MoE design keeps the active compute low. At 3.8B active parameters per forward pass (despite 25.2B total weights), the inference cost per token batch is closer to a 4B dense model than a 26B dense model. This is why the VRAM requirement is manageable: you need room to store the 25.2B weights plus activation memory, but the compute per step is light.

Benchmark Results

Be honest with yourself about these numbers before deploying. All figures in this section are from Google’s DiffusionGemma model card, comparing DiffusionGemma 26B-A4B against the Gemma 4 26B-A4B model it’s built on.

Reasoning and Knowledge

The AIME gap is the most telling: 69.1% vs 88.3%. On competition math, the quality cost of diffusion-based generation is significant. Gemma 4 is clearly better for tasks requiring multi-step reasoning.

Coding

Google’s model card shows DiffusionGemma trailing Gemma 4 on both code-generation (LiveCodeBench v6) and competitive-programming (Codeforces ELO) benchmarks — an 8-point gap on LiveCodeBench and a nearly 300-point gap in Codeforces rating. For anything beyond simple code generation or syntax correction, expect the quality difference to show up in production.

Multimodal

The OmniDocBench number is the interesting exception: DiffusionGemma outperforms Gemma 4 on document parsing. OCR, layout-aware extraction, and dense document understanding are tasks where the bidirectional attention in the diffusion decoder provides a structural advantage. If document processing is your primary workload, this score is worth taking seriously.

Inference Speed

These numbers are for low-batch-size inference. Single-request latency depends on the diffusion pass count, which varies with output length and content complexity.

Hardware Requirements

The BF16 and FP8 figures aren’t published directly by Google — they’re standard back-of-envelope math from the disclosed 25.2B total parameter count (roughly 2 bytes/parameter for BF16, ~1 byte/parameter plus overhead for FP8), not an official spec. Treat them as planning estimates and verify against your own inference stack before committing hardware.

The NVFP4 path is the practical consumer-GPU option. An RTX 5090 has 32GB of VRAM, comfortably fitting the ~18GB quantized model. RTX 4090 (24GB) is marginal — you may be able to run quantized inference, but with limited headroom for context.

NVIDIA has published an NVFP4-quantized model (nvidia/diffusiongemma-26B-A4B-it-NVFP4) built with NVIDIA Model Optimizer, targeting NVIDIA Hopper (H100) and Blackwell hardware.

Deployment

vLLM (recommended for production)

DiffusionGemma is the first diffusion LLM (dLLM) with native, day-zero support in vLLM.

pip install vllm

vllm serve "google/diffusiongemma-26B-A4B-it"

Once running, the server exposes an OpenAI-compatible endpoint. Any client that uses the openai Python library can redirect with two env variable changes:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="google/diffusiongemma-26B-A4B-it",
    messages=[{"role": "user", "content": "Summarize this document..."}],
    max_tokens=1024
)

Transformers (for experimentation)

from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor

MODEL_ID = "google/diffusiongemma-26B-A4B-it"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto",
)

messages = [{"role": "user", "content": "Why is the sky blue?"}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)

output = model.generate(**inputs, max_new_tokens=512)
text = processor.decode(output[0], skip_special_tokens=False)

Note: DiffusionGemmaForBlockDiffusion is a new class specific to this model. Standard AutoModelForCausalLM will not work — this model’s generation loop is different from autoregressive models.

Image input

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://example.com/document.jpg"},
            {"type": "text", "text": "Extract all text from this document."}
        ]
    },
]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)

Image placement matters: Google’s documentation is explicit — “for optimal performance with multimodal inputs, place image content before the text in your prompt."

Token budgets for image resolution: the supported token budgets are 70, 140, 280, 560, and 1120. Lower budgets (70–280) for classification and captioning; higher budgets (560–1120) for OCR and document parsing where detail matters.

NVIDIA NIM container (for NVIDIA-managed deployment)

NVIDIA publishes a ready-to-run NIM container for DiffusionGemma:

export NIM_IMAGE_PATH="nvcr.io/nim/google/diffusiongemma-26b-a4b-it:latest"
docker run --gpus=all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  ${NIM_IMAGE_PATH}

For the NVFP4 checkpoint specifically, pull nvidia/diffusiongemma-26B-A4B-it-NVFP4 directly from Hugging Face and serve it with vLLM or Transformers as shown above.

Diffusion Sampling Settings

The model has configurable denoising parameters. Google’s recommended defaults:

Adaptive stopping kicks in when per-canvas entropy drops below 0.005 and predictions stabilize. For short factual responses, this typically terminates well before 48 steps. For longer, more complex outputs, expect the full step count.

You can trade quality for speed by reducing max_denoising_steps — fewer passes means faster generation but more noise in the output. Experiment on your specific workload to find the right balance.

Thinking Mode

DiffusionGemma includes a thinking mode, triggered via a <|think|> system-prompt token, that produces internal reasoning before a final answer:

# Enable thinking with system prompt
messages = [
    {"role": "system", "content": "<|think|>Work through the problem step by step before answering."},
    {"role": "user", "content": "What's the most efficient way to process 10,000 PDFs?"}
]

Thinking output is wrapped in <|channel>thought\n[reasoning]<channel|> markers, followed by the final answer. The reasoning token stream is separate from the output and can be surfaced to users or discarded.

Important: thinking mode adds denoising passes for the reasoning canvas before the answer canvas. It will slow down overall generation relative to non-thinking mode. For high-throughput use cases, profile carefully before enabling thinking by default.

When to Use DiffusionGemma (and When Not To)

Strong use cases

Document extraction and OCR. The OmniDocBench 1.5 result (0.319 vs Gemma 4’s 0.149) is the clearest signal that DiffusionGemma has a genuine advantage in document-parsing workflows. If you’re processing PDFs, invoices, forms, or dense structured documents, the combination of speed and this specific quality advantage makes it worth evaluating seriously.

High-throughput batch inference. Summarizing thousands of short texts, generating product descriptions at scale, or any batch workflow where throughput matters more than frontier reasoning quality. At 1100+ tokens/second on a single H100, you’re covering substantially more ground per GPU-hour.

Interactive applications requiring real-time streaming. The parallel generation means first-token latency is different from autoregressive models — but steady-state throughput is high. For chat-adjacent interfaces where users are watching tokens appear, the speed profile can feel noticeably snappier.

Multimodal pipelines with image/video input. The vision encoder handles object detection, document/PDF parsing, chart comprehension, OCR, and video processed as frames (up to 60 seconds at 1 fps). At 256K context, you can process a long document with many embedded images in a single inference call.

Cases where Gemma 4 is better

Multi-step reasoning. The AIME 69.1% vs 88.3% gap is not noise. If your workflow involves complex mathematical reasoning, extended logic chains, or any task that requires consistent multi-step coherence, use Gemma 4.

Coding. Google’s own model card shows an 8-point LiveCodeBench v6 gap (69.1% vs 77.1%) and a nearly 300-point Codeforces ELO gap (1429 vs 1718) relative to Gemma 4. For anything beyond simple code generation or syntax correction, the quality difference is likely to show in production.

Research or analysis tasks. BigBench Extra Hard (47.6% vs 64.8%) suggests that tasks requiring extended, structured reasoning will produce meaningfully lower quality outputs.

Comparison to Alternatives

Limitations to Know Before Deploying

Training cutoff is January 2025. The model’s world knowledge stops there. For tasks requiring up-to-date information, you need retrieval augmentation.

Canvas size constrains coherence. Each 256-token canvas is denoised semi-independently. Very long structured outputs — particularly those requiring strict formatting or referencing content from many pages back — can show incoherence at canvas boundaries. This is an architectural constraint, not something you can tune away.

Diffusion != autoregressive for all tasks. The generation mechanics change what “temperature” and “sampling” mean. Some prompting techniques that work reliably for autoregressive models (few-shot examples, specific chain-of-thought formats) behave differently under text diffusion. Plan for a re-evaluation pass if you’re porting an existing prompt library.

Experimental positioning. Google’s own developer guide describes DiffusionGemma as “this experimental model," and its announcement post recommends deploying standard Gemma 4 for applications that demand maximum quality. That’s a meaningful signal from the team that built it.

Builder Checklist

• Confirm your use case: speed-critical (batch/throughput) OR document extraction → evaluate DiffusionGemma; reasoning/coding → use Gemma 4
• Hardware path: NVFP4 for RTX 5090 (700+ tok/s), FP8 for H100 (1100+ tok/s), BF16 for multi-GPU research
• Use DiffusionGemmaForBlockDiffusion class in Transformers — standard AutoModelForCausalLM will not work
• Place images before text in multimodal prompts
• Set image token budget based on detail level: 70–280 for captioning/classification, 560–1120 for OCR/documents
• Benchmark your specific workload — don’t rely only on MMLU numbers for your decision
• Profile thinking mode overhead before enabling by default in high-throughput pipelines
• Plan for canvas boundary effects in very long structured outputs
• Implement retrieval augmentation if your task requires post-January-2025 world knowledge
• Review Apache 2.0 license terms for your commercial deployment (permissive; confirm with legal for regulated industries)

Model weights: google/diffusiongemma-26B-A4B-it NVFP4 variant: nvidia/diffusiongemma-26B-A4B-it-NVFP4 Official announcement: blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/ Google AI developer docs: ai.google.dev/gemma/docs/diffusiongemma Developer guide: developers.googleblog.com/diffusiongemma-the-developer-guide NVIDIA build endpoint: build.nvidia.com/google/diffusiongemma-26b-a4b-it