Speculative Decoding

In the rapidly evolving landscape of locally hosted Large Language Models (LLMs), performance often remains the primary bottleneck. While quantization and efficient kernel implementations have made running powerful models on consumer hardware a reality, a significant efficiency gap remains. This gap is bridged by speculative decoding, an advanced inference technique that allows local LLMs to operate with significantly greater throughput.

This post provides a technical summary of how speculative decoding works, the core mechanism, best practices for choosing a draft model, and the setup required for various tooling environments.

What is Speculative Decoding?

At its heart, speculative decoding is an attempt to reduce the computational cost of generating one token by speculating on several future tokens simultaneously.

In a traditional autoregressive generation process, the large target model must perform a full forward pass for every single token it outputs. Speculative decoding avoids this costly step by introducing a lightweight draft model.

How the Process Works:

1. Drafting: A smaller, faster "draft model" (e.g., a 0.5B parameter model) generates a sequence of several plausible next tokens (typically 4 to 8 tokens) based on the current context.
2. Verification: The larger, target model then processes this entire sequence of draft tokens in a single parallel forward pass. This is the crucial step where the true probability is computed for each candidate token.
3. Accept/Reject: The system compares the draft model's prediction against the target model's calculated distribution. If the draft token matches the target's distribution, it is accepted. If a mismatch occurs, the process pauses, and the first rejected token is corrected using the large model's own true probability distribution.
4. Iteration: The accepted prefix becomes the new context, and the draft process begins again.

  flowchart TD
    Context([Current Context]) --> Draft

    %% subgraph DraftPhase["Drafting Phase"]
        Draft[Small Draft Model<br/>Generates 4–8 plausible tokens]
    %% end

    Draft --> Verify

    %% subgraph VerifyPhase["Verification Phase"]
        Verify[Large Target Model<br/>Evaluates sequence in a single parallel pass]
    %% end

    Verify --> Check

    subgraph DecisionPhase["Decision"]
        Check{Compare Distributions}
        Accept[Accept valid draft tokens]
        Correct[Pause & correct first rejected token<br/>using Target Model distribution]
    end

    Check -- ✅ --> Accept
    Check -- 🛑 --> Correct

    Accept --> Iteration
    Correct --> Iteration

    subgraph NextStep["Iteration"]
        Iteration[New accepted prefix<br/>becomes updated context]
    end

    Iteration --> Context

    %% Styling & Colors
    classDef contextStyle fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#01579b;
    classDef draftStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#1b5e20;
    classDef verifyStyle fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#e65100;
    classDef decisionStyle fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#4a148c;
    classDef actionStyle fill:#efebe9,stroke:#4e342e,stroke-width:2px,color:#3e2723;

    class Context contextStyle;
    class Draft draftStyle;
    class Verify verifyStyle;
    class Check decisionStyle;
    class Accept,Correct actionStyle;
    class Iteration contextStyle;

The resulting speed-up is achieved because the verification cost for a sequence of tokens is roughly the same as generating a single token. A successful process means the acceptance rate multiplied by the draft length yields a net speed-up.

Technical Note: This technique is a lossless speed-up; the final output distribution remains identical to what the original, slow model would have produced.

Tooling & Setup

For developers running LLMs locally, implementing speculative decoding depends heavily on the inference framework being used. The setup requires two key components: the target model (the large, high-quality model) and the draft model (the small, fast companion).

llama.cpp

Requires explicit flags (-md) to link the two model files:

$ llama-server -m target.gguf -md draft.gguf -c 8192

vLLM

Requires programmatic initialization, passing both models to the LLM constructor.

from vllm import LLM; 

llm = LLM(model="target", speculative_model="draft")

Ollama

Support is available in recent builds via an environment variable:

$ OLLAMA_SPECULATIVE_MODEL="qwen3:0.6b" ollama serve

Ollama self-enables it on MLX environments. Framework-specific behavior may override manual setup. 

Choosing Your Models

The performance gains are not guaranteed; they rely heavily on selecting compatible models. The draft model cannot be a random placeholder; it must maintain coherence with the target.

Best Practices for Draft Model Selection:

• Size: The draft model should ideally be significantly smaller—often 1/4 to 1/10 the parameters of the target model.
• Architecture: It is critical that the draft and target models share the same underlying architecture (e.g., both Qwen3 family) to keep the token distributions consistent.
• Serialization: The tokenizer and vocabulary of the draft model must be identical to the target model; otherwise, the feature silently no-ops.

When It Helps and When It Fails

While the speedups cited can range from 1.5× to 3× on robust GPU setups, the benefits are highly contextual.

✅ When Speculative Decoding Excels:

• Agent Loops: When a system requires multiple short generation steps (as seen in advanced agent workflows), the accumulated gains from decoding at each step can dramatically improve the overall loop time.
• Long Context/RAG: Generating large volumes of sequential tokens (e.g., summarizing a long document or responding to a RAG query) is where the verification process saves the most time.

❌ When You Should Be Cautious:

• CPU-Only Systems: On CPU-only machines, the primary bottleneck is memory bandwidth, which largely neutralizes the benefit of speculative decoding.
• Poor Matching: If the draft/target pair is poorly matched, or if the required generation is extremely short (where the verification overhead outweighs the saved steps), the gain will be minimal.

🛑 Architectural Constraint

S.H.O.D.A.N. and Pydantic AI SDK

The S.H.O.D.A.N. Project prioritizes robust, industrial-grade stability over bleeding-edge feature adoption. While the concept of speculative decoding is a powerful performance feature, current implementations within high-level SDKs, such as the Pydantic AI SDK, do not offer native, stable support.

The Pydantic AI SDK is designed to be a simplified wrapper, managing the interface to the LLM (prompting, tool-calling), rather than managing the low-level inference mechanics required by speculative decoding. Implementing this feature would require complex pipeline control (draft generation, rejection sampling) that is currently outside the SDK's core abstraction layer.

For the immediate and near-term development cycle of S.H.O.D.A.N., we will not be utilizing speculative decoding. Instead, our architectural strategy involves interfacing with underlying inference engines (like Ollama or vLLM) through their respective command-line or specialized APIs, allowing us to manually manage the configuration needed for maximum performance. This ensures our workflow stability remains decoupled from the limitations of any single, high-level abstraction layer.

This arcticle and the illustrations were created using AI tools and agents.