opensource/developer-roadmap
3
0
Fork 0
mirror of https://github.com/kamranahmedse/developer-roadmap.git synced 2026-10-25 09:34:00 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
inference-engineering-resources
developer-roadmap/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md
T
Javier CanalesandClaude Opus 5.5 4f7e267d15 feat(inference-engineering): add learning resources to roadmap topics 
Add curated papers, official docs, open-source repos and books to 64
topics of the Inference Engineering roadmap, following the content
style guide (typed links, ordered by type, max 8 per topic).

Also normalise the heading on the Post-Training Quantization and
Multi-Cloud topics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 13:24:34 +02:00

721 B
Raw Permalink Blame History

Speculative Decoding

Speculative decoding takes advantage of spare compute during memory-bound decode by generating multiple draft tokens and validating them in a single forward pass. If the target model accepts N draft tokens, the forward pass produces N+1 tokens instead of one. The performance gain depends on draft token cost, draft sequence length, and acceptance rate, which decreases at higher temperatures and larger batch sizes.

Visit the following resources to learn more:

Reference in New Issue View Git Blame Copy Permalink