feat(inference-engineering): add learning resources to roadmap topics (#10406)

Add curated papers, official docs, open-source repos and books to 64 topics of the Inference Engineering roadmap, following the content style guide (typed links, ordered by type, max 8 per topic). Also normalise the heading on the Post-Training Quantization and Multi-Cloud topics. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 01:08:07 +08:00 · 2026-10-01 13:28:17 +02:00
parent 1e31884c0f
commit a670d7fbee
64 changed files with 390 additions and 66 deletions
@@ -1,3 +1,7 @@
 # Attention Mechanism
 
-Attention is the mechanism transformers use to relate each token to all previous tokens via query, key, and value projections. During decode, each new token attends to every prior token, and the KV cache stores precomputed keys and values to avoid recomputation. Attention is the most expensive and most heavily optimized operation in LLM inference.
+Attention is the mechanism transformers use to relate each token to all previous tokens via query, key, and value projections. During decode, each new token attends to every prior token, and the KV cache stores precomputed keys and values to avoid recomputation. Attention is the most expensive and most heavily optimized operation in LLM inference.
+
+Visit the following resources to learn more:
+
+- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762)
@@ -1,3 +1,10 @@
 # Attention Variants
 
-Newer attention algorithms reduce the quadratic time complexity of standard attention. Sliding window attention limits each token to attending to the nearest w tokens, turning O(N²) into O(Nw). Linear attention approximates the softmax with a linear-time kernel. Compressed attention periodically compresses earlier context. These variants trade off some quality for better scaling on long sequences.
+Newer attention algorithms reduce the quadratic time complexity of standard attention. Sliding window attention limits each token to attending to the nearest w tokens, turning O(N²) into O(Nw). Linear attention approximates the softmax with a linear-time kernel. Compressed attention periodically compresses earlier context. These variants trade off some quality for better scaling on long sequences.
+
+Visit the following resources to learn more:
+
+- [@article@Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150)
+- [@article@Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451)
+- [@article@Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://arxiv.org/abs/2108.12409)
+- [@article@Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)
@@ -1,3 +1,8 @@
 # Automatic Speech Recognition (ASR)
 
-ASR models like Whisper take audio input and produce text transcripts. Whisper is an encoder-decoder model where the decoder, an autoregressive transformer, dominates inference time. TensorRT-LLM with in-flight batching for the decoder delivers the best performance. For real-time transcription, streaming via WebSockets with VAD-segmented chunks achieves round-trip targets of around 200 milliseconds per chunk on optimized deployments.
+ASR models like Whisper take audio input and produce text transcripts. Whisper is an encoder-decoder model where the decoder, an autoregressive transformer, dominates inference time. TensorRT-LLM with in-flight batching for the decoder delivers the best performance. For real-time transcription, streaming via WebSockets with VAD-segmented chunks achieves round-trip targets of around 200 milliseconds per chunk on optimized deployments.
+
+Visit the following resources to learn more:
+
+- [@official@OpenAI Whisper Models on Hugging Face](https://huggingface.co/openai)
+- [@article@Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)](https://arxiv.org/abs/2212.04356)
@@ -1,3 +1,9 @@
 # Autoscaling
 
-Autoscaling dynamically adjusts the number of active model replicas to match incoming traffic, maintaining latency SLAs while minimizing idle GPU spend. Kubernetes is the standard orchestration layer, scaling replicas within a cluster based on traffic signals and utilization metrics. Key configuration parameters are min replicas, max replicas, the autoscaling window, the scale-down delay, and the concurrency target per replica.
+Autoscaling dynamically adjusts the number of active model replicas to match incoming traffic, maintaining latency SLAs while minimizing idle GPU spend. Kubernetes is the standard orchestration layer, scaling replicas within a cluster based on traffic signals and utilization metrics. Key configuration parameters are min replicas, max replicas, the autoscaling window, the scale-down delay, and the concurrency target per replica.
+
+Visit the following resources to learn more:
+
+- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
+- [@book@Designing Data-Intensive Applications](https://dataintensive.net/)
+- [@book@Site Reliability Engineering](https://sre.google/books/)
@@ -1,3 +1,8 @@
 # BERT-style Models

-BERT-style encoder-only models are the traditional architecture for text embeddings, producing a fixed-size vector by pooling the final hidden states of a bidirectional transformer. They are compact, fast, and well-suited for high-throughput indexing workloads where embedding quality is sufficient for the task. Models like BGE-small and all-MiniLM run efficiently on CPU or fractional GPU instances, making them the lowest-cost embedding option.
+BERT-style encoder-only models are the traditional architecture for text embeddings, producing a fixed-size vector by pooling the final hidden states of a bidirectional transformer. They are compact, fast, and well-suited for high-throughput indexing workloads where embedding quality is sufficient for the task. Models like BGE-small and all-MiniLM run efficiently on CPU or fractional GPU instances, making them the lowest-cost embedding option.
+
+Visit the following resources to learn more:
+
+- [@article@BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)
+- [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
@@ -1,3 +1,7 @@
 # Blackwell
 
-The Blackwell architecture, released in late 2024 with the B200 and B300, added FP4 Tensor Cores and new microscaling number formats including MXFP8, MXFP4, and NVFP4. B200 GPUs offer approximately 5 petaFLOPS of FP8 compute and up to 8 TB/s of memory bandwidth. They are the current gold standard for demanding inference workloads including large LLMs and video generation.
+The Blackwell architecture, released in late 2024 with the B200 and B300, added FP4 Tensor Cores and new microscaling number formats including MXFP8, MXFP4, and NVFP4. B200 GPUs offer approximately 5 petaFLOPS of FP8 compute and up to 8 TB/s of memory bandwidth. They are the current gold standard for demanding inference workloads including large LLMs and video generation.
+
+Visit the following resources to learn more:
+
+- [@official@NVIDIA Blackwell Architecture Technical Brief](https://resources.nvidia.com/en-us-blackwell-architecture)
@@ -1,3 +1,8 @@
 # Caching Approaches
 
-KV cache reuse is the primary caching strategy in LLM inference, covering both prefix caching (reusing cached KV values across requests that share a common prefix) and KV cache offloading (moving less-used cache blocks to slower storage tiers to free GPU VRAM). A related concern is cache-aware routing, which directs requests to replicas most likely to already hold the relevant cached prefix.
+KV cache reuse is the primary caching strategy in LLM inference, covering both prefix caching (reusing cached KV values across requests that share a common prefix) and KV cache offloading (moving less-used cache blocks to slower storage tiers to free GPU VRAM). A related concern is cache-aware routing, which directs requests to replicas most likely to already hold the relevant cached prefix.
+
+Visit the following resources to learn more:
+
+- [@opensource@LMCache](https://github.com/LMCache/LMCache)
+- [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444)
@@ -1,3 +1,7 @@
 # Containerization
 
-Containerization packages an inference service and all its dependencies into a Docker image that runs consistently across environments. Inference containers include the CUDA toolkit, Python packages, the inference engine, and any system libraries like ffmpeg for audio processing. Using a pinned dependency tree prevents breaking changes from affecting running services and ensures reproducible builds.
+Containerization packages an inference service and all its dependencies into a Docker image that runs consistently across environments. Inference containers include the CUDA toolkit, Python packages, the inference engine, and any system libraries like ffmpeg for audio processing. Using a pinned dependency tree prevents breaking changes from affecting running services and ensures reproducible builds.
+
+Visit the following resources to learn more:
+
+- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/)
@@ -1,3 +1,9 @@
 # CUDA
 
-CUDA is NVIDIA's proprietary platform for programming GPUs, consisting of a kernel language (C++), a driver for hardware management, and a runtime API for launching kernels. Every inference engine, framework, and library in the NVIDIA ecosystem depends on CUDA. Understanding CUDA is the foundation for diagnosing performance problems that tools and engines cannot solve automatically.
+CUDA is NVIDIA's proprietary platform for programming GPUs, consisting of a kernel language (C++), a driver for hardware management, and a runtime API for launching kernels. Every inference engine, framework, and library in the NVIDIA ecosystem depends on CUDA. Understanding CUDA is the foundation for diagnosing performance problems that tools and engines cannot solve automatically.
+
+Visit the following resources to learn more:
+
+- [@official@CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/)
+- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/)
+- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example)
@@ -1,3 +1,7 @@
 # Dependency Management
 
-Inference dependency chains are long and fragile due to the tight coupling between CUDA versions, framework versions, and hardware architectures. Pinning exact versions of every dependency prevents unexpected breakage when upstream packages release updates. Containers for newly released models often initially depend on pre-release software builds and need to be rebuilt on stable releases within days of the model launch.
+Inference dependency chains are long and fragile due to the tight coupling between CUDA versions, framework versions, and hardware architectures. Pinning exact versions of every dependency prevents unexpected breakage when upstream packages release updates. Containers for newly released models often initially depend on pre-release software builds and need to be rebuilt on stable releases within days of the model launch.
+
+Visit the following resources to learn more:
+
+- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/)
@@ -1,3 +1,7 @@
 # Desktop Inference
 
-Desktop inference runs on consumer hardware, primarily Apple M-series unified memory systems and high-end NVIDIA consumer GPUs. Apple hardware offers very large unified memory capacity at moderate bandwidth, enabling larger models than discrete consumer GPUs; NVIDIA consumer GPUs offer higher bandwidth with smaller capacity. Tools like Ollama and llama.cpp make it practical to run aggressively quantized models on consumer hardware with minimal setup.
+Desktop inference runs on consumer hardware, primarily Apple M-series unified memory systems and high-end NVIDIA consumer GPUs. Apple hardware offers very large unified memory capacity at moderate bandwidth, enabling larger models than discrete consumer GPUs; NVIDIA consumer GPUs offer higher bandwidth with smaller capacity. Tools like Ollama and llama.cpp make it practical to run aggressively quantized models on consumer hardware with minimal setup.
+
+Visit the following resources to learn more:
+
+- [@opensource@ComfyUI](https://github.com/comfyanonymous/ComfyUI)
@@ -1,3 +1,10 @@
 # Diffusion Models
 
-Diffusion models generate images by starting from random noise and iteratively denoising it over 30 to 50 steps, guided by a text prompt. Each step runs the denoiser twice, once with conditioning and once without, to apply classifier-free guidance. The entire process operates in a low-dimensional latent space rather than full pixel space to make attention computationally feasible.
+Diffusion models generate images by starting from random noise and iteratively denoising it over 30 to 50 steps, guided by a text prompt. Each step runs the denoiser twice, once with conditioning and once without, to apply classifier-free guidance. The entire process operates in a low-dimensional latent space rather than full pixel space to make attention computationally feasible.
+
+Visit the following resources to learn more:
+
+- [@article@Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)
+- [@article@High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752)
+- [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952)
+- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748)
@@ -1,3 +1,7 @@
 # Disaggregation
 
-Disaggregation separates the prefill and decode phases onto independent GPU instances that can scale separately. The prefill engine processes the input sequence and sends the resulting KV cache to the decode engine via the hardware interconnect. This removes the resource contention between compute-heavy prefill and memory-heavy decode. Disaggregation is most effective at high traffic volumes, for large models, and with prefill-heavy workloads like code editors.
+Disaggregation separates the prefill and decode phases onto independent GPU instances that can scale separately. The prefill engine processes the input sequence and sends the resulting KV cache to the decode engine via the hardware interconnect. This removes the resource contention between compute-heavy prefill and memory-heavy decode. Disaggregation is most effective at high traffic volumes, for large models, and with prefill-heavy workloads like code editors.
+
+Visit the following resources to learn more:
+
+- [@article@Beyond the Buzz: A Pragmatic Take on Inference Disaggregation](https://arxiv.org/abs/2506.05508)
@@ -1,3 +1,7 @@
 # Distillation
 
-Distillation trains a smaller student model to emulate a larger teacher model by exposing it to the teacher's probability distributions, not just its final outputs. Unlike fine-tuning on synthetic data, distillation transfers the teacher's reasoning behavior. The resulting student model is smaller and faster while retaining much of the teacher's quality on the same tasks.
+Distillation trains a smaller student model to emulate a larger teacher model by exposing it to the teacher's probability distributions, not just its final outputs. Unlike fine-tuning on synthetic data, distillation transfers the teacher's reasoning behavior. The resulting student model is smaller and faster while retaining much of the teacher's quality on the same tasks.
+
+Visit the following resources to learn more:
+
+- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042)
@@ -1,3 +1,7 @@
 # Draft-Target Speculative Decoding
 
-Draft-target speculation uses a small draft model, typically at least ten times smaller than the target, to generate draft tokens for the target model to verify. The draft model should share the same tokenizer and general behavior as the target, often being a smaller member of the same model family. It is the simplest speculation method to set up but introduces the most overhead due to running a separate model.
+Draft-target speculation uses a small draft model, typically at least ten times smaller than the target, to generate draft tokens for the target model to verify. The draft model should share the same tokenizer and general behavior as the target, often being a smaller member of the same model family. It is the simplest speculation method to set up but introduces the most overhead due to running a separate model.
+
+Visit the following resources to learn more:
+
+- [@article@Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192)
@@ -1,3 +1,9 @@
 # EAGLE
 
-EAGLE is a purpose-built speculation method where a small draft model is trained to accept hidden states from the target model as input. By seeing the target's internal representations, EAGLE achieves higher acceptance rates and longer draft sequences (up to eight tokens) than using a general-purpose small model. It integrates into the same forward pass as the target model, eliminating the orchestration overhead of separate draft and target model runs.
+EAGLE is a purpose-built speculation method where a small draft model is trained to accept hidden states from the target model as input. By seeing the target's internal representations, EAGLE achieves higher acceptance rates and longer draft sequences (up to eight tokens) than using a general-purpose small model. It integrates into the same forward pass as the target model, eliminating the orchestration overhead of separate draft and target model runs.
+
+Visit the following resources to learn more:
+
+- [@article@EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://arxiv.org/abs/2401.15077)
+- [@article@EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://arxiv.org/abs/2406.16858)
+- [@article@EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://arxiv.org/abs/2503.01840)
@@ -1,3 +1,10 @@
 # Embedding Models
 
-Embedding models convert variable-length text or image input into a fixed-length vector that encodes semantic meaning. These vectors are used in RAG, search, recommendation, and agent memory systems. Embedding inference has two distinct traffic profiles: high-throughput backfills for bulk indexing and low-latency lookups for real-time user queries, which are best served by separate deployments.
+Embedding models convert variable-length text or image input into a fixed-length vector that encodes semantic meaning. These vectors are used in RAG, search, recommendation, and agent memory systems. Embedding inference has two distinct traffic profiles: high-throughput backfills for bulk indexing and low-latency lookups for real-time user queries, which are best served by separate deployments.
+
+Visit the following resources to learn more:
+
+- [@article@BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)
+- [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
+- [@article@Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147)
+- [@article@MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316)
@@ -1,3 +1,8 @@
 # Expert Parallelism
 
-Expert Parallelism distributes MoE experts across GPUs so each GPU hosts multiple complete experts. Tokens are routed to the GPU holding their activated experts, and each expert runs independently within a single GPU without splitting across multiple. EP has lower inter-GPU communication overhead than Tensor Parallelism and scales well to multi-node deployments via InfiniBand.
+Expert Parallelism distributes MoE experts across GPUs so each GPU hosts multiple complete experts. Tokens are routed to the GPU holding their activated experts, and each expert runs independently within a single GPU without splitting across multiple. EP has lower inter-GPU communication overhead than Tensor Parallelism and scales well to multi-node deployments via InfiniBand.
+
+Visit the following resources to learn more:
+
+- [@opensource@NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
+- [@article@Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053)
@@ -1,3 +1,7 @@
 # Few-Step Image Generation

-Few-step image generation models produce usable images in eight or fewer denoising steps by training the model to collapse the standard 30 to 50 step process into a fraction of the iterations. Techniques include latent consistency distillation and adversarial training, both of which teach the model to reach high-quality outputs in fewer passes. They are a strong fit for latency-sensitive use cases like real-time filters, but produce noticeably lower quality than full-step models and are a model selection decision rather than an inference optimization technique.
+Few-step image generation models produce usable images in eight or fewer denoising steps by training the model to collapse the standard 30 to 50 step process into a fraction of the iterations. Techniques include latent consistency distillation and adversarial training, both of which teach the model to reach high-quality outputs in fewer passes. They are a strong fit for latency-sensitive use cases like real-time filters, but produce noticeably lower quality than full-step models and are a model selection decision rather than an inference optimization technique.
+
+Visit the following resources to learn more:
+
+- [@article@Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378)
@@ -1,3 +1,7 @@
 # Fine-Tuning
 
-Fine-tuning takes a pretrained foundation model and adapts it to a specific domain by training on additional labeled data. A small fine-tuned model can match a much larger general model on constrained tasks like text-to-SQL. Smaller models are faster and cheaper to serve, so fine-tuning is one of the highest-leverage tools for hitting latency and cost targets.
+Fine-tuning takes a pretrained foundation model and adapts it to a specific domain by training on additional labeled data. A small fine-tuned model can match a much larger general model on constrained tasks like text-to-SQL. Smaller models are faster and cheaper to serve, so fine-tuning is one of the highest-leverage tools for hitting latency and cost targets.
+
+Visit the following resources to learn more:
+
+- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042)
@@ -1,3 +1,10 @@
 # FlashAttention
 
-FlashAttention is a series of hand-optimized CUDA kernels that compute attention with fewer reads and writes to GPU memory by fusing the attention algorithm into a single kernel. Standard attention stores intermediate matrices to memory between steps; FlashAttention eliminates those round-trips. FlashAttention 3 targets Hopper GPUs and FlashAttention 4 targets Blackwell GPUs.
+FlashAttention is a series of hand-optimized CUDA kernels that compute attention with fewer reads and writes to GPU memory by fusing the attention algorithm into a single kernel. Standard attention stores intermediate matrices to memory between steps; FlashAttention eliminates those round-trips. FlashAttention 3 targets Hopper GPUs and FlashAttention 4 targets Blackwell GPUs.
+
+Visit the following resources to learn more:
+
+- [@opensource@Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention)
+- [@article@FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135)
+- [@article@FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://arxiv.org/abs/2307.08691)
+- [@article@FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://arxiv.org/abs/2407.08608)
@@ -1,3 +1,9 @@
 # GPU Architecture
 
-GPU architecture determines the hard limits of what inference can achieve on a given device. This group covers compute, memory, interconnects, and sizing — the four dimensions an inference engineer needs to understand to make informed decisions about hardware selection, parallelism strategy, and optimization priorities.
+GPU architecture determines the hard limits of what inference can achieve on a given device. This group covers compute, memory, interconnects, and sizing — the four dimensions an inference engineer needs to understand to make informed decisions about hardware selection, parallelism strategy, and optimization priorities.
+
+Visit the following resources to learn more:
+
+- [@article@NVIDIA Tesla: A Unified Graphics and Computing Architecture](https://ieeexplore.ieee.org/document/4523358)
+- [@article@GPU Glossary](https://modal.com/gpu-glossary)
+- [@book@Programming Massively Parallel Processors](https://www.goodreads.com/work/editions/10244675)
@@ -1,3 +1,7 @@
 # Grace and Vera CPUs
 
-NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink Chip-to-Chip at up to 900 GB/s, far higher than standard PCIe connections. This high-bandwidth CPU-GPU link makes offloading KV caches and LoRA weights to CPU memory practical, since data can be retrieved quickly enough to remain on the critical path. Grace pairs with Hopper and Blackwell; Vera pairs with the upcoming Rubin architecture.
+NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink Chip-to-Chip at up to 900 GB/s, far higher than standard PCIe connections. This high-bandwidth CPU-GPU link makes offloading KV caches and LoRA weights to CPU memory practical, since data can be retrieved quickly enough to remain on the critical path. Grace pairs with Hopper and Blackwell; Vera pairs with the upcoming Rubin architecture.
+
+Visit the following resources to learn more:
+
+- [@official@NVIDIA Grace CPU](https://www.nvidia.com/en-us/data-center/grace-cpu/)
@@ -1,3 +1,7 @@
 # Hopper
 
-The Hopper architecture, released in 2022 with the H100 and later the H200, introduced FP8 Tensor Cores offering twice the compute throughput of FP16 and asynchronous data transfer features used by FlashAttention 3. H100 and H200 GPUs remain among the most widely deployed inference accelerators due to their maturity, broad software support, and relative availability compared to newer generations.
+The Hopper architecture, released in 2022 with the H100 and later the H200, introduced FP8 Tensor Cores offering twice the compute throughput of FP16 and asynchronous data transfer features used by FlashAttention 3. H100 and H200 GPUs remain among the most widely deployed inference accelerators due to their maturity, broad software support, and relative availability compared to newer generations.
+
+Visit the following resources to learn more:
+
+- [@official@NVIDIA H100 Tensor Core GPU Architecture](https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c)
@@ -1,3 +1,10 @@
 # Image Generation Architecture
 
-Image generation models are pipelines of multiple components: a text encoder that converts the prompt into conditioning, a denoising model that iteratively refines a latent representation from noise into an image, and a variational autoencoder that converts the final latent to pixels. The denoising model is the largest and most compute-intensive component and runs up to 100 forward passes per image generation.
+Image generation models are pipelines of multiple components: a text encoder that converts the prompt into conditioning, a denoising model that iteratively refines a latent representation from noise into an image, and a variational autoencoder that converts the final latent to pixels. The denoising model is the largest and most compute-intensive component and runs up to 100 forward passes per image generation.
+
+Visit the following resources to learn more:
+
+- [@article@Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)
+- [@article@High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752)
+- [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952)
+- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748)
@@ -1,3 +1,8 @@
 # Image Generation Inference
 
-Production image generation inference uses SGLang Diffusion, TensorRT, or hand-optimized PyTorch. The most important optimization target is the attention kernel, where FlashAttention 3 or 4 outperforms the default on Hopper and Blackwell respectively. GEMM kernels for linear layers can be quantized to FP8 safely. Torch compilation with manual kernel plugins caches the compiled engine to avoid multi-minute recompilation on every cold start.
+Production image generation inference uses SGLang Diffusion, TensorRT, or hand-optimized PyTorch. The most important optimization target is the attention kernel, where FlashAttention 3 or 4 outperforms the default on Hopper and Blackwell respectively. GEMM kernels for linear layers can be quantized to FP8 safely. Torch compilation with manual kernel plugins caches the compiled engine to avoid multi-minute recompilation on every cold start.
+
+Visit the following resources to learn more:
+
+- [@opensource@ComfyUI](https://github.com/comfyanonymous/ComfyUI)
+- [@article@Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378)
@@ -1,3 +1,7 @@
 # Inference vs. Training
 
-Training is the process of learning model weights from data. Inference is serving those weights in production to generate outputs. Training runs are compute-intensive batch jobs; inference is a continuous, latency-sensitive service. Where training success is measured in loss curves, inference success is measured in time to first token, tokens per second, and uptime.
+Training is the process of learning model weights from data. Inference is serving those weights in production to generate outputs. Training runs are compute-intensive batch jobs; inference is a continuous, latency-sensitive service. Where training success is measured in loss curves, inference success is measured in time to first token, tokens per second, and uptime.
+
+Visit the following resources to learn more:
+
+- [@article@Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
@@ -1,3 +1,8 @@
 # Instances and Interconnects
 
-Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, and NVSwitch creates an all-to-all fabric across all eight GPUs. Between nodes, InfiniBand provides inter-node connectivity at substantially lower bandwidth than NVLink. Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated slices, each with dedicated compute and memory, allowing small models to run on fractions of a high-end GPU without resource contention.
+Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, and NVSwitch creates an all-to-all fabric across all eight GPUs. Between nodes, InfiniBand provides inter-node connectivity at substantially lower bandwidth than NVLink. Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated slices, each with dedicated compute and memory, allowing small models to run on fractions of a high-end GPU without resource contention.
+
+Visit the following resources to learn more:
+
+- [@official@NVLink and NVSwitch](https://www.nvidia.com/en-us/data-center/nvlink/)
+- [@official@NVIDIA InfiniBand](https://www.nvidia.com/en-us/networking/products/infiniband/)
@@ -1,3 +1,8 @@
 # KV Cache Storage
 
-KV caches can be stored in GPU VRAM (fastest, tens to hundreds of gigabytes), CPU RAM (slower, hundreds of gigabytes to terabytes), local SSD, or networked SSD. Moving less frequently accessed KV blocks to slower storage frees VRAM for active requests. NVIDIA Dynamo's KV Block Manager provides APIs for managing KV blocks across these storage tiers transparently.
+KV caches can be stored in GPU VRAM (fastest, tens to hundreds of gigabytes), CPU RAM (slower, hundreds of gigabytes to terabytes), local SSD, or networked SSD. Moving less frequently accessed KV blocks to slower storage frees VRAM for active requests. NVIDIA Dynamo's KV Block Manager provides APIs for managing KV blocks across these storage tiers transparently.
+
+Visit the following resources to learn more:
+
+- [@opensource@LMCache](https://github.com/LMCache/LMCache)
+- [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444)
@@ -1,3 +1,7 @@
 # KV Cache
 
-The KV cache stores the key and value tensors computed during attention for each previously processed token. Without it, attention would require recomputing every prior token on each decode step, making inference prohibitively slow. The KV cache lives in GPU memory and its size grows with sequence length and batch size, making it a major target for memory management techniques.
+The KV cache stores the key and value tensors computed during attention for each previously processed token. Without it, attention would require recomputing every prior token on each decode step, making inference prohibitively slow. The KV cache lives in GPU memory and its size grows with sequence length and batch size, making it a major target for memory management techniques.
+
+Visit the following resources to learn more:
+
+- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
@@ -1,3 +1,7 @@
 # Large Language Models

-An LLM is a decoder-only transformer with three main components: an embedding layer that converts input tokens to vectors, a stack of transformer blocks that process those vectors, and an output language modeling head that converts the final hidden states into a probability distribution over the vocabulary. The configuration file for an open model specifies all architectural details including layer count, hidden dimensions, and attention heads.
+An LLM is a decoder-only transformer with three main components: an embedding layer that converts input tokens to vectors, a stack of transformer blocks that process those vectors, and an output language modeling head that converts the final hidden states into a probability distribution over the vocabulary. The configuration file for an open model specifies all architectural details including layer count, hidden dimensions, and attention heads.
+
+Visit the following resources to learn more:
+
+- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch)
@@ -1,3 +1,8 @@
 # LLM Inference Bottlenecks
 
-LLM prefill is compute-bound because it processes the entire input sequence as one large matrix multiplication, performing many operations per byte loaded. LLM decode is memory-bound because it generates tokens one at a time via vector-matrix multiplications that require loading all model weights from memory for each token while performing relatively few arithmetic operations.
+LLM prefill is compute-bound because it processes the entire input sequence as one large matrix multiplication, performing many operations per byte loaded. LLM decode is memory-bound because it generates tokens one at a time via vector-matrix multiplications that require loading all model weights from memory for each token while performing relatively few arithmetic operations.
+
+Visit the following resources to learn more:
+
+- [@article@GPU Glossary](https://modal.com/gpu-glossary)
+- [@book@AI Systems Performance Engineering](https://www.oreilly.com/library/view/ai-systems-performance/9781098170202/)
@@ -1,3 +1,8 @@
 # Long Context Handling
 
-Long sequences produce KV caches large enough to strain GPU memory, slow down decode, and increase prefill time. Techniques for handling long context include FlashAttention for memory-efficient attention computation, PagedAttention for non-contiguous KV storage, chunked prefill to interleave long prefill with decode, and model-level approaches like sliding window attention that limit how many prior tokens each token attends to.
+Long sequences produce KV caches large enough to strain GPU memory, slow down decode, and increase prefill time. Techniques for handling long context include FlashAttention for memory-efficient attention computation, PagedAttention for non-contiguous KV storage, chunked prefill to interleave long prefill with decode, and model-level approaches like sliding window attention that limit how many prior tokens each token attends to.
+
+Visit the following resources to learn more:
+
+- [@article@Ring Attention with Blockwise Transformers for Near-Infinite Context](https://arxiv.org/abs/2310.01889)
+- [@article@RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864)
@@ -1,3 +1,7 @@
 # Mixture of Experts (MoE)
 
-Mixture of Experts models replace dense linear layers with a set of smaller expert matrices, activating only a subset of experts per token via a learned router. A model like Qwen3-235B activates only 22 billion of its 235 billion parameters per forward pass, making single-user inference efficient. In batched production serving, different tokens activate different experts, so nearly all parameters are active across the batch.
+Mixture of Experts models replace dense linear layers with a set of smaller expert matrices, activating only a subset of experts per token via a learned router. A model like Qwen3-235B activates only 22 billion of its 235 billion parameters per forward pass, making single-user inference efficient. In batched production serving, different tokens activate different experts, so nearly all parameters are active across the batch.
+
+Visit the following resources to learn more:
+
+- [@article@Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538)
@@ -1,3 +1,8 @@
 # Model Evaluation
 
-Model evaluation, or evals, is the practice of systematically measuring model quality for a specific task or domain. Standard intelligence benchmarks like MMLU measure general capabilities but can be gamed; product-specific evals that test real use cases provide more reliable signal. High-confidence evals are a prerequisite before investing in inference optimization, ensuring the model is actually useful before making it fast.
+Model evaluation, or evals, is the practice of systematically measuring model quality for a specific task or domain. Standard intelligence benchmarks like MMLU measure general capabilities but can be gamed; product-specific evals that test real use cases provide more reliable signal. High-confidence evals are a prerequisite before investing in inference optimization, ensuring the model is actually useful before making it fast.
+
+Visit the following resources to learn more:
+
+- [@article@How to Fine-Tune Qwen3 to GPT-4o Level Performance](https://ghost.oxen.ai/how-to-fine-tune-qwen3-to-gpt-4o-level-performance/)
+- [@book@Evals for AI Engineers](https://www.oreilly.com/library/view/evals-for-ai/9798341660717/)
@@ -1,3 +1,7 @@
 # Model Selection
 
-The most important decision in model performance optimization is which model to use. Smaller models with fewer parameters are faster and cheaper to serve than larger ones. The goal is to find the smallest model that meets quality requirements for the task. Model architecture also determines which inference engines and optimization techniques are available, so popular architectures with broad tooling support are preferable.
+The most important decision in model performance optimization is which model to use. Smaller models with fewer parameters are faster and cheaper to serve than larger ones. The goal is to find the smallest model that meets quality requirements for the task. Model architecture also determines which inference engines and optimization techniques are available, so popular architectures with broad tooling support are preferable.
+
+Visit the following resources to learn more:
+
+- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/)
@@ -1,3 +1,10 @@
-#  Multi-Cloud
+# Multi-Cloud
 
-Multi-cloud inference distributes GPU capacity across providers and regions to maximize availability, minimize latency, and meet data residency requirements. This group covers procurement strategy, capacity management, geographic routing, and the reliability patterns that make large-scale inference resilient to hardware and provider failures.
+Multi-cloud inference distributes GPU capacity across providers and regions to maximize availability, minimize latency, and meet data residency requirements. This group covers procurement strategy, capacity management, geographic routing, and the reliability patterns that make large-scale inference resilient to hardware and provider failures.
+
+Visit the following resources to learn more:
+
+- [@article@SemiAnalysis](https://semianalysis.com/)
+- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
+- [@book@Designing Data-Intensive Applications](https://dataintensive.net/)
+- [@book@Site Reliability Engineering](https://sre.google/books/)
@@ -1,3 +1,7 @@
 # N-gram Speculation
 
-N-gram speculation builds a dictionary during prefill that maps token prefixes to observed n-gram continuations from the input. During decode, generated tokens are looked up in this dictionary and matching suffixes become draft tokens with no separate model required. It is most effective for code completion, where output frequently mirrors input syntax, and can produce very long draft sequences with high acceptance rates in that domain.
+N-gram speculation builds a dictionary during prefill that maps token prefixes to observed n-gram continuations from the input. During decode, generated tokens are looked up in this dictionary and matching suffixes become draft tokens with no separate model required. It is most effective for code completion, where output frequently mirrors input syntax, and can produce very long draft sequences with high acceptance rates in that domain.
+
+Visit the following resources to learn more:
+
+- [@article@Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://arxiv.org/abs/2402.02057)
@@ -1,3 +1,9 @@
 # Neural Networks
 
-A neural network is a layered structure of nodes where each node applies a learned weight matrix and bias to its input. Multiple layers connected in sequence form the foundation for all generative AI models. Inference engineers need a working model of how neural networks compute outputs to understand where performance bottlenecks arise.
+A neural network is a layered structure of nodes where each node applies a learned weight matrix and bias to its input. Multiple layers connected in sequence form the foundation for all generative AI models. Inference engineers need a working model of how neural networks compute outputs to understand where performance bottlenecks arise.
+
+Visit the following resources to learn more:
+
+- [@book@Deep Learning](https://www.deeplearningbook.org/)
+- [@book@Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python-second-edition)
+- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch)
@@ -1,3 +1,7 @@
 # NVIDIA Dynamo
 
-Dynamo supports conditional disaggregation, where requests are first sent to the decode engine and only routed to a prefill engine when the input is long or not already cached. The ratio of prefill to decode engines is configurable at runtime and can shift as traffic patterns change. This dynamic allocation prevents the prefill queue from becoming a bottleneck and avoids wasting decode engine resources on prefill-heavy traffic.
+Dynamo supports conditional disaggregation, where requests are first sent to the decode engine and only routed to a prefill engine when the input is long or not already cached. The ratio of prefill to decode engines is configurable at runtime and can shift as traffic patterns change. This dynamic allocation prevents the prefill queue from becoming a bottleneck and avoids wasting decode engine resources on prefill-heavy traffic.
+
+Visit the following resources to learn more:
+
+- [@official@NVIDIA Dynamo Documentation](https://docs.nvidia.com/dynamo/latest/index.html)
@@ -1,3 +1,7 @@
 # NVIDIA Dynamo
 
-Dynamo is NVIDIA's open-source distributed serving platform that sits above inference engines and provides orchestration for large-scale deployments. It adds KV-aware routing, disaggregated prefill/decode serving, multi-node parallelism, and an SLA-based planner that dynamically adjusts the number of prefill and decode workers based on TTFT and TPS targets.
+Dynamo is NVIDIA's open-source distributed serving platform that sits above inference engines and provides orchestration for large-scale deployments. It adds KV-aware routing, disaggregated prefill/decode serving, multi-node parallelism, and an SLA-based planner that dynamically adjusts the number of prefill and decode workers based on TTFT and TPS targets.
+
+Visit the following resources to learn more:
+
+- [@official@NVIDIA Dynamo Documentation](https://docs.nvidia.com/dynamo/latest/index.html)
@@ -1,3 +1,7 @@
 # Observability
 
-Inference observability tracks total request volume, input and output sequence lengths, response codes, latency percentiles (P50, P90, P99 of TTFT, TPS, and end-to-end latency), replica count, GPU utilization, and queue depth. These metrics are interdependent; a latency spike could come from long input sequences, high concurrency, or a hardware issue. Observability must integrate with existing tooling like Grafana, Datadog, and PagerDuty to put inference metrics in context.
+Inference observability tracks total request volume, input and output sequence lengths, response codes, latency percentiles (P50, P90, P99 of TTFT, TPS, and end-to-end latency), replica count, GPU utilization, and queue depth. These metrics are interdependent; a latency spike could come from long input sequences, high concurrency, or a hardware issue. Observability must integrate with existing tooling like Grafana, Datadog, and PagerDuty to put inference metrics in context.
+
+Visit the following resources to learn more:
+
+- [@official@NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems)
@@ -1,3 +1,7 @@
 # ONNX Runtime
 
-ONNX Runtime is an open-source inference runtime that executes models exported to the ONNX format, which stores both weights and a computation graph for cross-platform portability. It supports multiple hardware backends including CUDA, DirectML, and CPU. It is widely used for embedding models, vision models, and pipelines that need to run across hardware environments without engine-specific configuration.
+ONNX Runtime is an open-source inference runtime that executes models exported to the ONNX format, which stores both weights and a computation graph for cross-platform portability. It supports multiple hardware backends including CUDA, DirectML, and CPU. It is widely used for embedding models, vision models, and pipelines that need to run across hardware environments without engine-specific configuration.
+
+Visit the following resources to learn more:
+
+- [@official@ONNX Runtime](https://onnxruntime.ai/)
@@ -1,3 +1,7 @@
 # Open vs. Closed Models
 
-Closed models are accessed exclusively through a vendor API; the weights are never released. Open models publish their weights for download and self-hosting. For inference engineers, this distinction determines the entire problem space: closed models require no infrastructure but offer no control over latency, cost, or configuration, while open models require building and operating a full inference stack but give complete control over every layer of it.
+Closed models are accessed exclusively through a vendor API; the weights are never released. Open models publish their weights for download and self-hosting. For inference engineers, this distinction determines the entire problem space: closed models require no infrastructure but offer no control over latency, cost, or configuration, while open models require building and operating a full inference stack but give complete control over every layer of it.
+
+Visit the following resources to learn more:
+
+- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/)
@@ -1,3 +1,7 @@
 # PagedAttention
 
-PagedAttention stores the KV cache in fixed-size pages rather than a single contiguous memory block, allowing it to be distributed across GPU memory without fragmentation. It uses a lookup table to access pages on demand, reducing memory waste from over-allocation and enabling more requests to share GPU memory. It is the memory management technique that made vLLM's high throughput possible.
+PagedAttention stores the KV cache in fixed-size pages rather than a single contiguous memory block, allowing it to be distributed across GPU memory without fragmentation. It uses a lookup table to access pages on demand, reducing memory waste from over-allocation and enabling more requests to share GPU memory. It is the memory management technique that made vLLM's high throughput possible.
+
+Visit the following resources to learn more:
+
+- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
@@ -1,3 +1,8 @@
 # Pipeline Parallelism
 
-Pipeline Parallelism splits an LLM's layers into sequential stages and assigns each stage to a different GPU or node. Unlike Tensor Parallelism, each GPU holds complete layers rather than layer fragments. It introduces pipeline bubbles where GPUs wait for preceding stages, reducing efficiency. It is used for multi-node inference with dense models, typically combined with Tensor Parallelism within each node.
+Pipeline Parallelism splits an LLM's layers into sequential stages and assigns each stage to a different GPU or node. Unlike Tensor Parallelism, each GPU holds complete layers rather than layer fragments. It introduces pipeline bubbles where GPUs wait for preceding stages, reducing efficiency. It is used for multi-node inference with dense models, typically combined with Tensor Parallelism within each node.
+
+Visit the following resources to learn more:
+
+- [@opensource@NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
+- [@article@Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053)
@@ -1,3 +1,12 @@
-Post-Training Quantization
+# Post-Training Quantization

-Post-training quantization converts finished model weights to lower precision after training is complete, using a small calibration dataset to compute per-layer scale factors that minimize the gap between quantized and original outputs. It is the standard approach for most deployments since it requires no retraining and can be applied to any existing model. The leading tool is NVIDIA ModelOpt, which exports quantized models in formats compatible with all major inference engines.
+Post-training quantization converts finished model weights to lower precision after training is complete, using a small calibration dataset to compute per-layer scale factors that minimize the gap between quantized and original outputs. It is the standard approach for most deployments since it requires no retraining and can be applied to any existing model. The leading tool is NVIDIA ModelOpt, which exports quantized models in formats compatible with all major inference engines.
+
+Visit the following resources to learn more:
+
+- [@opensource@AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://github.com/mit-han-lab/llm-awq)
+- [@opensource@bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
+- [@opensource@SmoothQuant](https://github.com/mit-han-lab/smoothquant)
+- [@article@GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323)
+- [@article@LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339)
+- [@article@SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/abs/2301.00774)
@@ -1,3 +1,8 @@
 # Profiling Performance
 
-Profiling reveals where time is spent within a benchmark, identifying which operations consume the most compute or memory. Tools include PyTorch Profiler, NVIDIA Nsight Systems for system-wide GPU and CPU analysis, and NVIDIA Nsight Compute for per-kernel analysis. Profiling is most useful when writing custom inference code or contributing to inference frameworks, rather than configuring existing engines.
+Profiling reveals where time is spent within a benchmark, identifying which operations consume the most compute or memory. Tools include PyTorch Profiler, NVIDIA Nsight Systems for system-wide GPU and CPU analysis, and NVIDIA Nsight Compute for per-kernel analysis. Profiling is most useful when writing custom inference code or contributing to inference frameworks, rather than configuring existing engines.
+
+Visit the following resources to learn more:
+
+- [@official@PyTorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html)
+- [@official@NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems)
@@ -1,3 +1,7 @@
 # PyTorch
 
-PyTorch is the standard Python framework for building and running neural networks. It abstracts CUDA into a tensor operation API and provides automatic differentiation for training. For inference, PyTorch supports torch.compile for automatic kernel optimization and serves as the foundation that all major inference engines build on top of.
+PyTorch is the standard Python framework for building and running neural networks. It abstracts CUDA into a tensor operation API and provides automatic differentiation for training. For inference, PyTorch supports torch.compile for automatic kernel optimization and serves as the foundation that all major inference engines build on top of.
+
+Visit the following resources to learn more:
+
+- [@official@PyTorch Performance Tuning Guide](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html)
@@ -1,3 +1,12 @@
 # Quantization
 
-Quantization converts model weights and other values from their native precision to a lower-precision format. Cutting precision in half doubles memory bandwidth for decode and enables twice the FLOPS for prefill on compatible Tensor Cores. The tradeoff is potential quality loss from reduced numerical precision, which can compound across the many calculations in a forward pass.
+Quantization converts model weights and other values from their native precision to a lower-precision format. Cutting precision in half doubles memory bandwidth for decode and enables twice the FLOPS for prefill on compatible Tensor Cores. The tradeoff is potential quality loss from reduced numerical precision, which can compound across the many calculations in a forward pass.
+
+Visit the following resources to learn more:
+
+- [@opensource@AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://github.com/mit-han-lab/llm-awq)
+- [@opensource@bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
+- [@opensource@SmoothQuant](https://github.com/mit-han-lab/smoothquant)
+- [@article@GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323)
+- [@article@LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339)
+- [@article@SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/abs/2301.00774)
@@ -1,3 +1,9 @@
 # Routing and Load Balancing
 
-Routers make request-level decisions about which replica to send a request to, considering factors like KV cache prefix match and LoRA availability. Load balancers make system-level decisions to distribute traffic evenly across replicas. In systems with prefix caching, naive round-robin load balancing misses cache opportunities; intelligent routing increases cache hit rates and reduces TTFT.
+Routers make request-level decisions about which replica to send a request to, considering factors like KV cache prefix match and LoRA availability. Load balancers make system-level decisions to distribute traffic evenly across replicas. In systems with prefix caching, naive round-robin load balancing misses cache opportunities; intelligent routing increases cache hit rates and reduces TTFT.
+
+Visit the following resources to learn more:
+
+- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
+- [@book@Designing Data-Intensive Applications](https://dataintensive.net/)
+- [@book@Site Reliability Engineering](https://sre.google/books/)
@@ -1,3 +1,7 @@
 # SGLang
 
-SGLang is a high-throughput inference engine with a flexible frontend language for customizing server behavior. It has strong support for MoE models like DeepSeek and Qwen and works closely with model labs to optimize new architectural features. SGLang Diffusion extends it to image and video generation models with pipeline abstractions and diffusion-specific parallelism.
+SGLang is a high-throughput inference engine with a flexible frontend language for customizing server behavior. It has strong support for MoE models like DeepSeek and Qwen and works closely with model labs to optimize new architectural features. SGLang Diffusion extends it to image and video generation models with pipeline abstractions and diffusion-specific parallelism.
+
+Visit the following resources to learn more:
+
+- [@opensource@sgl-project/sglang](https://github.com/sgl-project/sglang)
@@ -1,3 +1,9 @@
 # Sizing and Procurement
 
-Estimating GPU requirements starts with VRAM: roughly 1 GB per billion parameters at FP8 for weights, plus at least 50 percent headroom for the KV cache. This total is rounded up to the nearest available instance size. Cloud GPUs are available from hyperscalers like AWS and GCP, neoclouds like CoreWeave and Nebius, and spot markets; large deployments combine reserved instances for baseline load with on-demand and spot capacity for bursts.
+Estimating GPU requirements starts with VRAM: roughly 1 GB per billion parameters at FP8 for weights, plus at least 50 percent headroom for the KV cache. This total is rounded up to the nearest available instance size. Cloud GPUs are available from hyperscalers like AWS and GCP, neoclouds like CoreWeave and Nebius, and spot markets; large deployments combine reserved instances for baseline load with on-demand and spot capacity for bursts.
+
+Visit the following resources to learn more:
+
+- [@article@SemiAnalysis](https://semianalysis.com/)
+- [@book@Designing Data-Intensive Applications](https://dataintensive.net/)
+- [@book@Site Reliability Engineering](https://sre.google/books/)
@@ -1,3 +1,8 @@
 # Speculative Decoding
 
-Speculative decoding takes advantage of spare compute during memory-bound decode by generating multiple draft tokens and validating them in a single forward pass. If the target model accepts N draft tokens, the forward pass produces N+1 tokens instead of one. The performance gain depends on draft token cost, draft sequence length, and acceptance rate, which decreases at higher temperatures and larger batch sizes.
+Speculative decoding takes advantage of spare compute during memory-bound decode by generating multiple draft tokens and validating them in a single forward pass. If the target model accepts N draft tokens, the forward pass produces N+1 tokens instead of one. The performance gain depends on draft token cost, draft sequence length, and acceptance rate, which decreases at higher temperatures and larger batch sizes.
+
+Visit the following resources to learn more:
+
+- [@article@Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192)
+- [@article@Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://arxiv.org/abs/2401.10774)
@@ -1,3 +1,8 @@
 # Tensor Parallelism
 
-Tensor Parallelism splits each layer's weight matrices across multiple GPUs and distributes the matmul computation. Each GPU handles a shard of each layer and the results are combined via an all-reduce operation before the next layer. It reduces per-user latency by sharing both VRAM and compute across GPUs, but requires fast NVLink interconnects and is not well-suited for multi-node deployments.
+Tensor Parallelism splits each layer's weight matrices across multiple GPUs and distributes the matmul computation. Each GPU handles a shard of each layer and the results are combined via an all-reduce operation before the next layer. It reduces per-user latency by sharing both VRAM and compute across GPUs, but requires fast NVLink interconnects and is not well-suited for multi-node deployments.
+
+Visit the following resources to learn more:
+
+- [@opensource@NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
+- [@article@Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053)
@@ -1,3 +1,7 @@
 # TensorRT-LLM
 
-TensorRT-LLM is NVIDIA's inference engine combining PyTorch-based model execution with proprietary NVIDIA kernels unavailable in other engines. It achieves the highest performance among the three major engines, with excellent support for Hopper and Blackwell architectures and NVIDIA-specific formats like NVFP4. The tradeoff is a steeper learning curve and narrower model support compared to vLLM or SGLang.
+TensorRT-LLM is NVIDIA's inference engine combining PyTorch-based model execution with proprietary NVIDIA kernels unavailable in other engines. It achieves the highest performance among the three major engines, with excellent support for Hopper and Blackwell architectures and NVIDIA-specific formats like NVFP4. The tradeoff is a steeper learning curve and narrower model support compared to vLLM or SGLang.
+
+Visit the following resources to learn more:
+
+- [@opensource@NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)
@@ -1,3 +1,7 @@
 # TensorRT
 
-TensorRT is NVIDIA's high-performance inference runtime that compiles PyTorch or ONNX models into optimized engines using fused kernels, quantization, and hardware-specific tuning. It is the standard choice for image and video generation models and for any workload where maximum throughput on NVIDIA hardware is the primary goal. Compiled engines are cached to disk, eliminating recompilation overhead on subsequent cold starts.
+TensorRT is NVIDIA's high-performance inference runtime that compiles PyTorch or ONNX models into optimized engines using fused kernels, quantization, and hardware-specific tuning. It is the standard choice for image and video generation models and for any workload where maximum throughput on NVIDIA hardware is the primary goal. Compiled engines are cached to disk, eliminating recompilation overhead on subsequent cold starts.
+
+Visit the following resources to learn more:
+
+- [@official@TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/)
@@ -1,3 +1,7 @@
 # Transformer Blocks
 
-Transformer blocks are the repeated hidden layers of an LLM. Each block contains an attention sublayer, a feed-forward neural network sublayer, and normalization layers. The feed-forward network holds the majority of model parameters, while attention is the most computationally complex operation per token.
+Transformer blocks are the repeated hidden layers of an LLM. Each block contains an attention sublayer, a feed-forward neural network sublayer, and normalization layers. The feed-forward network holds the majority of model parameters, while attention is the most computationally complex operation per token.
+
+Visit the following resources to learn more:
+
+- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762)
@@ -1,3 +1,8 @@
 # Hugging Face Transformers and Diffusers
 
-The transformers and diffusers libraries provide reference implementations of popular LLM and image/video generation architectures. They are not designed for production serving but are essential for understanding model configurations, downloading weights, and prototyping inference. Model cards on Hugging Face use transformers or diffusers sample code as the canonical usage examples.
+The transformers and diffusers libraries provide reference implementations of popular LLM and image/video generation architectures. They are not designed for production serving but are essential for understanding model configurations, downloading weights, and prototyping inference. Model cards on Hugging Face use transformers or diffusers sample code as the canonical usage examples.
+
+Visit the following resources to learn more:
+
+- [@official@Hugging Face Diffusers](https://huggingface.co/docs/diffusers/index)
+- [@official@Hugging Face Transformers](https://huggingface.co/docs/transformers/index)
@@ -1,3 +1,9 @@
 # Video Generation Inference
 
-Video generation models run on full nodes of eight GPUs with a batch size of one because the latent space is too large to batch effectively. Attention over the three-dimensional video latent space consumes 70 to 80 percent of compute time and is the primary optimization target. Attention caching techniques that reuse outputs from previous timesteps or transformer layers can improve speed by 30 to 40 percent.
+Video generation models run on full nodes of eight GPUs with a batch size of one because the latent space is too large to batch effectively. Attention over the three-dimensional video latent space consumes 70 to 80 percent of compute time and is the primary optimization target. Attention caching techniques that reuse outputs from previous timesteps or transformer layers can improve speed by 30 to 40 percent.
+
+Visit the following resources to learn more:
+
+- [@opensource@Cache-DiT](https://github.com/vipshop/cache-dit)
+- [@opensource@TeaCache: Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://github.com/ali-vilab/TeaCache)
+- [@article@SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://arxiv.org/abs/2410.02367)
@@ -1,3 +1,8 @@
 # Video Generation Models
 
-Video generation models extend image generation to three-dimensional latent space, representing width, height, and time simultaneously. They run the same number of denoising steps as image models but process far more data per step. Attention over the full video latent space dominates compute time, making video generation the most resource-intensive common inference workload.
+Video generation models extend image generation to three-dimensional latent space, representing width, height, and time simultaneously. They run the same number of denoising steps as image models but process far more data per step. Attention over the full video latent space dominates compute time, making video generation the most resource-intensive common inference workload.
+
+Visit the following resources to learn more:
+
+- [@article@Video Diffusion Models](https://arxiv.org/abs/2204.03458)
+- [@article@Imagen Video: High Definition Video Generation with Diffusion Models](https://arxiv.org/abs/2210.02303)
@@ -1,3 +1,10 @@
 # Vision Language Models (VLMs)
 
-VLMs combine a large language model with a small vision encoder that converts input images or video into a sequence of visual tokens. A high-resolution image typically adds about 1,000 tokens to the input sequence. All LLM inference optimization techniques apply to VLMs, but longer sequences make quantization, prefix caching, and disaggregation especially important.
+VLMs combine a large language model with a small vision encoder that converts input images or video into a sequence of visual tokens. A high-resolution image typically adds about 1,000 tokens to the input sequence. All LLM inference optimization techniques apply to VLMs, but longer sequences make quantization, prefix caching, and disaggregation especially important.
+
+Visit the following resources to learn more:
+
+- [@article@BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597)
+- [@article@Visual Instruction Tuning](https://arxiv.org/abs/2304.08485)
+- [@article@Segment Anything](https://arxiv.org/abs/2304.02643)
+- [@article@SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://arxiv.org/abs/2509.11815)
@@ -1,3 +1,7 @@
 # vLLM
 
-vLLM is the most widely adopted LLM inference engine, offering broad support for model architectures, hardware platforms (NVIDIA, AMD, Google TPUs), and optimization techniques including prefix caching, quantization, speculative decoding, and parallelism. Its primary advantage is ease of adoption and the fastest day-zero support for newly released open models. It also supports multimodal inference via vLLM Omni.
+vLLM is the most widely adopted LLM inference engine, offering broad support for model architectures, hardware platforms (NVIDIA, AMD, Google TPUs), and optimization techniques including prefix caching, quantization, speculative decoding, and parallelism. Its primary advantage is ease of adoption and the fastest day-zero support for newly released open models. It also supports multimodal inference via vLLM Omni.
+
+Visit the following resources to learn more:
+
+- [@opensource@vllm-project/vllm](https://github.com/vllm-project/vllm)
@@ -1,3 +1,12 @@
 # Writing Custom Kernels
 
-Custom kernels are written when existing libraries do not fully utilize a GPU's capabilities for a specific operation or architecture. Most production kernels are written in C++ using CUTLASS and CuTe for GEMM-style operations and FlashInfer for attention variants; Triton offers a Python-based alternative with a lower barrier to entry. Writing an effective kernel requires understanding the target GPU's memory hierarchy, SM occupancy, and Tensor Core instruction set — skills that sit at the intersection of software and hardware.
+Custom kernels are written when existing libraries do not fully utilize a GPU's capabilities for a specific operation or architecture. Most production kernels are written in C++ using CUTLASS and CuTe for GEMM-style operations and FlashInfer for attention variants; Triton offers a Python-based alternative with a lower barrier to entry. Writing an effective kernel requires understanding the target GPU's memory hierarchy, SM occupancy, and Tensor Core instruction set — skills that sit at the intersection of software and hardware.
+
+Visit the following resources to learn more:
+
+- [@official@CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/)
+- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/)
+- [@opensource@NVIDIA/cutlass](https://github.com/NVIDIA/cutlass)
+- [@opensource@deepseek-ai/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)
+- [@article@FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://arxiv.org/abs/2501.01005)
+- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example)