Skip to content

T5Gemma 2 Speculators

Diffusion speculators for an encoder–decoder model

T5Gemma 2 Speculators

Trained DFlash, DFlare, and DSpark for T5Gemma 2 1B–1B and built a standalone vLLM serving plugin. A speculator proposes a block of tokens for the base model to verify, reducing the number of sequential generation steps.

  • Speculative decoding
  • DFlash / DFlare / DSpark
  • vLLM
GitHub · vLLM plugin Hugging Face
1.55× HumanEval · DSpark Large BS1 · K=7 1.47× HumanEval · DSpark Base BS4 · K=7 200k Examples in the data mixture 6 datasets · 6 epochs

What I built

  • Adapted Speculators training to T5Gemma 2 with online hidden-state extraction. The frozen base model and trainable draft shared one RTX 5070 Ti 16 GB; the data mixture used a 90% train / 10% validation split.
  • Trained five-layer Large drafts and a three-layer DSpark Base. The plugin adds encoder–decoder attention, encoder K/V caching, and CUDA Graph support. Weights, training commands, optimizer states, logs, and benchmarks are published on Hugging Face.

Key results

DSpark Large reached 1.553× on HumanEval (BS1, K=7). DSpark Base reached 1.467× on HumanEval and 1.399× on MBPP (BS4, K=7); at K=3 it accelerated all six evaluated datasets at both BS1 and BS4. DFlare Large reached 1.222× on HumanEval (BS1, K=7), but often slowed generation down; DFlash Large was slower than the base model throughout this benchmark campaign.

Measured in vLLM on an RTX 5070 Ti 16 GB / WSL2 with greedy decoding, CUDA Graph, and a 128-token output limit. BS is request concurrency; K is the number of proposed tokens. Speedup compares end-to-end tokens/s against the base model in the same campaign. Base and Large were measured separately; gains depend on the workload and settings.

Benchmark tables and settings

Training code · exact commits

Original papers