diff --git a/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md b/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md index 690ebb8d5..801149e68 100644 --- a/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md +++ b/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md @@ -1,3 +1,7 @@ # Attention Mechanism -Attention is the mechanism transformers use to relate each token to all previous tokens via query, key, and value projections. During decode, each new token attends to every prior token, and the KV cache stores precomputed keys and values to avoid recomputation. Attention is the most expensive and most heavily optimized operation in LLM inference. \ No newline at end of file +Attention is the mechanism transformers use to relate each token to all previous tokens via query, key, and value projections. During decode, each new token attends to every prior token, and the KV cache stores precomputed keys and values to avoid recomputation. Attention is the most expensive and most heavily optimized operation in LLM inference. + +Visit the following resources to learn more: + +- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md b/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md index 630466a14..f3aa78434 100644 --- a/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md +++ b/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md @@ -1,3 +1,10 @@ # Attention Variants -Newer attention algorithms reduce the quadratic time complexity of standard attention. Sliding window attention limits each token to attending to the nearest w tokens, turning O(N²) into O(Nw). Linear attention approximates the softmax with a linear-time kernel. Compressed attention periodically compresses earlier context. These variants trade off some quality for better scaling on long sequences. \ No newline at end of file +Newer attention algorithms reduce the quadratic time complexity of standard attention. Sliding window attention limits each token to attending to the nearest w tokens, turning O(N²) into O(Nw). Linear attention approximates the softmax with a linear-time kernel. Compressed attention periodically compresses earlier context. These variants trade off some quality for better scaling on long sequences. + +Visit the following resources to learn more: + +- [@article@Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) +- [@article@Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) +- [@article@Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://arxiv.org/abs/2108.12409) +- [@article@Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md b/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md index f6c51eaea..db87f0d5a 100644 --- a/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md +++ b/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md @@ -1,3 +1,8 @@ # Automatic Speech Recognition (ASR) -ASR models like Whisper take audio input and produce text transcripts. Whisper is an encoder-decoder model where the decoder, an autoregressive transformer, dominates inference time. TensorRT-LLM with in-flight batching for the decoder delivers the best performance. For real-time transcription, streaming via WebSockets with VAD-segmented chunks achieves round-trip targets of around 200 milliseconds per chunk on optimized deployments. \ No newline at end of file +ASR models like Whisper take audio input and produce text transcripts. Whisper is an encoder-decoder model where the decoder, an autoregressive transformer, dominates inference time. TensorRT-LLM with in-flight batching for the decoder delivers the best performance. For real-time transcription, streaming via WebSockets with VAD-segmented chunks achieves round-trip targets of around 200 milliseconds per chunk on optimized deployments. + +Visit the following resources to learn more: + +- [@official@OpenAI Whisper Models on Hugging Face](https://huggingface.co/openai) +- [@article@Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)](https://arxiv.org/abs/2212.04356) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md b/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md index 94757738d..f387ee650 100644 --- a/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md +++ b/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md @@ -1,3 +1,9 @@ # Autoscaling -Autoscaling dynamically adjusts the number of active model replicas to match incoming traffic, maintaining latency SLAs while minimizing idle GPU spend. Kubernetes is the standard orchestration layer, scaling replicas within a cluster based on traffic signals and utilization metrics. Key configuration parameters are min replicas, max replicas, the autoscaling window, the scale-down delay, and the concurrency target per replica. \ No newline at end of file +Autoscaling dynamically adjusts the number of active model replicas to match incoming traffic, maintaining latency SLAs while minimizing idle GPU spend. Kubernetes is the standard orchestration layer, scaling replicas within a cluster based on traffic signals and utilization metrics. Key configuration parameters are min replicas, max replicas, the autoscaling window, the scale-down delay, and the concurrency target per replica. + +Visit the following resources to learn more: + +- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) +- [@book@Designing Data-Intensive Applications](https://dataintensive.net/) +- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md b/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md index adb912cc8..0e2705e90 100644 --- a/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md +++ b/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md @@ -1,3 +1,8 @@ # BERT-style Models -BERT-style encoder-only models are the traditional architecture for text embeddings, producing a fixed-size vector by pooling the final hidden states of a bidirectional transformer. They are compact, fast, and well-suited for high-throughput indexing workloads where embedding quality is sufficient for the task. Models like BGE-small and all-MiniLM run efficiently on CPU or fractional GPU instances, making them the lowest-cost embedding option. \ No newline at end of file +BERT-style encoder-only models are the traditional architecture for text embeddings, producing a fixed-size vector by pooling the final hidden states of a bidirectional transformer. They are compact, fast, and well-suited for high-throughput indexing workloads where embedding quality is sufficient for the task. Models like BGE-small and all-MiniLM run efficiently on CPU or fractional GPU instances, making them the lowest-cost embedding option. + +Visit the following resources to learn more: + +- [@article@BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) +- [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md b/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md index 6f1509663..fcb57a33b 100644 --- a/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md +++ b/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md @@ -1,3 +1,7 @@ # Blackwell -The Blackwell architecture, released in late 2024 with the B200 and B300, added FP4 Tensor Cores and new microscaling number formats including MXFP8, MXFP4, and NVFP4. B200 GPUs offer approximately 5 petaFLOPS of FP8 compute and up to 8 TB/s of memory bandwidth. They are the current gold standard for demanding inference workloads including large LLMs and video generation. \ No newline at end of file +The Blackwell architecture, released in late 2024 with the B200 and B300, added FP4 Tensor Cores and new microscaling number formats including MXFP8, MXFP4, and NVFP4. B200 GPUs offer approximately 5 petaFLOPS of FP8 compute and up to 8 TB/s of memory bandwidth. They are the current gold standard for demanding inference workloads including large LLMs and video generation. + +Visit the following resources to learn more: + +- [@official@NVIDIA Blackwell Architecture Technical Brief](https://resources.nvidia.com/en-us-blackwell-architecture) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md b/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md index 7a1ffdee1..10959d4d9 100644 --- a/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md +++ b/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md @@ -1,3 +1,8 @@ # Caching Approaches -KV cache reuse is the primary caching strategy in LLM inference, covering both prefix caching (reusing cached KV values across requests that share a common prefix) and KV cache offloading (moving less-used cache blocks to slower storage tiers to free GPU VRAM). A related concern is cache-aware routing, which directs requests to replicas most likely to already hold the relevant cached prefix. \ No newline at end of file +KV cache reuse is the primary caching strategy in LLM inference, covering both prefix caching (reusing cached KV values across requests that share a common prefix) and KV cache offloading (moving less-used cache blocks to slower storage tiers to free GPU VRAM). A related concern is cache-aware routing, which directs requests to replicas most likely to already hold the relevant cached prefix. + +Visit the following resources to learn more: + +- [@opensource@LMCache](https://github.com/LMCache/LMCache) +- [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md b/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md index af056fc81..050dc2c49 100644 --- a/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md +++ b/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md @@ -1,3 +1,7 @@ # Containerization -Containerization packages an inference service and all its dependencies into a Docker image that runs consistently across environments. Inference containers include the CUDA toolkit, Python packages, the inference engine, and any system libraries like ffmpeg for audio processing. Using a pinned dependency tree prevents breaking changes from affecting running services and ensures reproducible builds. \ No newline at end of file +Containerization packages an inference service and all its dependencies into a Docker image that runs consistently across environments. Inference containers include the CUDA toolkit, Python packages, the inference engine, and any system libraries like ffmpeg for audio processing. Using a pinned dependency tree prevents breaking changes from affecting running services and ensures reproducible builds. + +Visit the following resources to learn more: + +- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md b/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md index 54cf7ecbf..fb437281b 100644 --- a/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md +++ b/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md @@ -1,3 +1,9 @@ # CUDA -CUDA is NVIDIA's proprietary platform for programming GPUs, consisting of a kernel language (C++), a driver for hardware management, and a runtime API for launching kernels. Every inference engine, framework, and library in the NVIDIA ecosystem depends on CUDA. Understanding CUDA is the foundation for diagnosing performance problems that tools and engines cannot solve automatically. \ No newline at end of file +CUDA is NVIDIA's proprietary platform for programming GPUs, consisting of a kernel language (C++), a driver for hardware management, and a runtime API for launching kernels. Every inference engine, framework, and library in the NVIDIA ecosystem depends on CUDA. Understanding CUDA is the foundation for diagnosing performance problems that tools and engines cannot solve automatically. + +Visit the following resources to learn more: + +- [@official@CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/) +- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/) +- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md b/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md index 6d2d14816..cf672ef34 100644 --- a/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md +++ b/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md @@ -1,3 +1,7 @@ # Dependency Management -Inference dependency chains are long and fragile due to the tight coupling between CUDA versions, framework versions, and hardware architectures. Pinning exact versions of every dependency prevents unexpected breakage when upstream packages release updates. Containers for newly released models often initially depend on pre-release software builds and need to be rebuilt on stable releases within days of the model launch. \ No newline at end of file +Inference dependency chains are long and fragile due to the tight coupling between CUDA versions, framework versions, and hardware architectures. Pinning exact versions of every dependency prevents unexpected breakage when upstream packages release updates. Containers for newly released models often initially depend on pre-release software builds and need to be rebuilt on stable releases within days of the model launch. + +Visit the following resources to learn more: + +- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md index d2846bde7..c309427bc 100644 --- a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md +++ b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md @@ -1,3 +1,7 @@ # Desktop Inference -Desktop inference runs on consumer hardware, primarily Apple M-series unified memory systems and high-end NVIDIA consumer GPUs. Apple hardware offers very large unified memory capacity at moderate bandwidth, enabling larger models than discrete consumer GPUs; NVIDIA consumer GPUs offer higher bandwidth with smaller capacity. Tools like Ollama and llama.cpp make it practical to run aggressively quantized models on consumer hardware with minimal setup. \ No newline at end of file +Desktop inference runs on consumer hardware, primarily Apple M-series unified memory systems and high-end NVIDIA consumer GPUs. Apple hardware offers very large unified memory capacity at moderate bandwidth, enabling larger models than discrete consumer GPUs; NVIDIA consumer GPUs offer higher bandwidth with smaller capacity. Tools like Ollama and llama.cpp make it practical to run aggressively quantized models on consumer hardware with minimal setup. + +Visit the following resources to learn more: + +- [@opensource@ComfyUI](https://github.com/comfyanonymous/ComfyUI) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md index e28f6c80d..d9b7a49a2 100644 --- a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md +++ b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md @@ -1,3 +1,10 @@ # Diffusion Models -Diffusion models generate images by starting from random noise and iteratively denoising it over 30 to 50 steps, guided by a text prompt. Each step runs the denoiser twice, once with conditioning and once without, to apply classifier-free guidance. The entire process operates in a low-dimensional latent space rather than full pixel space to make attention computationally feasible. \ No newline at end of file +Diffusion models generate images by starting from random noise and iteratively denoising it over 30 to 50 steps, guided by a text prompt. Each step runs the denoiser twice, once with conditioning and once without, to apply classifier-free guidance. The entire process operates in a low-dimensional latent space rather than full pixel space to make attention computationally feasible. + +Visit the following resources to learn more: + +- [@article@Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) +- [@article@High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) +- [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) +- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md b/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md index cbfbeefd6..2ea1e4cdc 100644 --- a/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md +++ b/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md @@ -1,3 +1,7 @@ # Disaggregation -Disaggregation separates the prefill and decode phases onto independent GPU instances that can scale separately. The prefill engine processes the input sequence and sends the resulting KV cache to the decode engine via the hardware interconnect. This removes the resource contention between compute-heavy prefill and memory-heavy decode. Disaggregation is most effective at high traffic volumes, for large models, and with prefill-heavy workloads like code editors. \ No newline at end of file +Disaggregation separates the prefill and decode phases onto independent GPU instances that can scale separately. The prefill engine processes the input sequence and sends the resulting KV cache to the decode engine via the hardware interconnect. This removes the resource contention between compute-heavy prefill and memory-heavy decode. Disaggregation is most effective at high traffic volumes, for large models, and with prefill-heavy workloads like code editors. + +Visit the following resources to learn more: + +- [@article@Beyond the Buzz: A Pragmatic Take on Inference Disaggregation](https://arxiv.org/abs/2506.05508) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md b/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md index ffe8dae24..76082df12 100644 --- a/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md +++ b/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md @@ -1,3 +1,7 @@ # Distillation -Distillation trains a smaller student model to emulate a larger teacher model by exposing it to the teacher's probability distributions, not just its final outputs. Unlike fine-tuning on synthetic data, distillation transfers the teacher's reasoning behavior. The resulting student model is smaller and faster while retaining much of the teacher's quality on the same tasks. \ No newline at end of file +Distillation trains a smaller student model to emulate a larger teacher model by exposing it to the teacher's probability distributions, not just its final outputs. Unlike fine-tuning on synthetic data, distillation transfers the teacher's reasoning behavior. The resulting student model is smaller and faster while retaining much of the teacher's quality on the same tasks. + +Visit the following resources to learn more: + +- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md b/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md index 3b5bdd4fc..f5c20953f 100644 --- a/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md +++ b/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md @@ -1,3 +1,7 @@ # Draft-Target Speculative Decoding -Draft-target speculation uses a small draft model, typically at least ten times smaller than the target, to generate draft tokens for the target model to verify. The draft model should share the same tokenizer and general behavior as the target, often being a smaller member of the same model family. It is the simplest speculation method to set up but introduces the most overhead due to running a separate model. \ No newline at end of file +Draft-target speculation uses a small draft model, typically at least ten times smaller than the target, to generate draft tokens for the target model to verify. The draft model should share the same tokenizer and general behavior as the target, often being a smaller member of the same model family. It is the simplest speculation method to set up but introduces the most overhead due to running a separate model. + +Visit the following resources to learn more: + +- [@article@Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md b/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md index 6524045c1..f64f3334a 100644 --- a/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md +++ b/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md @@ -1,3 +1,9 @@ # EAGLE -EAGLE is a purpose-built speculation method where a small draft model is trained to accept hidden states from the target model as input. By seeing the target's internal representations, EAGLE achieves higher acceptance rates and longer draft sequences (up to eight tokens) than using a general-purpose small model. It integrates into the same forward pass as the target model, eliminating the orchestration overhead of separate draft and target model runs. \ No newline at end of file +EAGLE is a purpose-built speculation method where a small draft model is trained to accept hidden states from the target model as input. By seeing the target's internal representations, EAGLE achieves higher acceptance rates and longer draft sequences (up to eight tokens) than using a general-purpose small model. It integrates into the same forward pass as the target model, eliminating the orchestration overhead of separate draft and target model runs. + +Visit the following resources to learn more: + +- [@article@EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://arxiv.org/abs/2401.15077) +- [@article@EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://arxiv.org/abs/2406.16858) +- [@article@EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://arxiv.org/abs/2503.01840) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md b/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md index ffde49cfe..5e057244f 100644 --- a/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md +++ b/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md @@ -1,3 +1,10 @@ # Embedding Models -Embedding models convert variable-length text or image input into a fixed-length vector that encodes semantic meaning. These vectors are used in RAG, search, recommendation, and agent memory systems. Embedding inference has two distinct traffic profiles: high-throughput backfills for bulk indexing and low-latency lookups for real-time user queries, which are best served by separate deployments. \ No newline at end of file +Embedding models convert variable-length text or image input into a fixed-length vector that encodes semantic meaning. These vectors are used in RAG, search, recommendation, and agent memory systems. Embedding inference has two distinct traffic profiles: high-throughput backfills for bulk indexing and low-latency lookups for real-time user queries, which are best served by separate deployments. + +Visit the following resources to learn more: + +- [@article@BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) +- [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084) +- [@article@Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147) +- [@article@MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md b/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md index 97b5ecf00..c4844f5f8 100644 --- a/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md +++ b/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md @@ -1,3 +1,8 @@ # Expert Parallelism -Expert Parallelism distributes MoE experts across GPUs so each GPU hosts multiple complete experts. Tokens are routed to the GPU holding their activated experts, and each expert runs independently within a single GPU without splitting across multiple. EP has lower inter-GPU communication overhead than Tensor Parallelism and scales well to multi-node deployments via InfiniBand. \ No newline at end of file +Expert Parallelism distributes MoE experts across GPUs so each GPU hosts multiple complete experts. Tokens are routed to the GPU holding their activated experts, and each expert runs independently within a single GPU without splitting across multiple. EP has lower inter-GPU communication overhead than Tensor Parallelism and scales well to multi-node deployments via InfiniBand. + +Visit the following resources to learn more: + +- [@opensource@NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM) +- [@article@Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md b/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md index 9517449ff..86272102e 100644 --- a/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md +++ b/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md @@ -1,3 +1,7 @@ # Few-Step Image Generation -Few-step image generation models produce usable images in eight or fewer denoising steps by training the model to collapse the standard 30 to 50 step process into a fraction of the iterations. Techniques include latent consistency distillation and adversarial training, both of which teach the model to reach high-quality outputs in fewer passes. They are a strong fit for latency-sensitive use cases like real-time filters, but produce noticeably lower quality than full-step models and are a model selection decision rather than an inference optimization technique. \ No newline at end of file +Few-step image generation models produce usable images in eight or fewer denoising steps by training the model to collapse the standard 30 to 50 step process into a fraction of the iterations. Techniques include latent consistency distillation and adversarial training, both of which teach the model to reach high-quality outputs in fewer passes. They are a strong fit for latency-sensitive use cases like real-time filters, but produce noticeably lower quality than full-step models and are a model selection decision rather than an inference optimization technique. + +Visit the following resources to learn more: + +- [@article@Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md b/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md index 86dc5d576..15d97e342 100644 --- a/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md +++ b/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md @@ -1,3 +1,7 @@ # Fine-Tuning -Fine-tuning takes a pretrained foundation model and adapts it to a specific domain by training on additional labeled data. A small fine-tuned model can match a much larger general model on constrained tasks like text-to-SQL. Smaller models are faster and cheaper to serve, so fine-tuning is one of the highest-leverage tools for hitting latency and cost targets. \ No newline at end of file +Fine-tuning takes a pretrained foundation model and adapts it to a specific domain by training on additional labeled data. A small fine-tuned model can match a much larger general model on constrained tasks like text-to-SQL. Smaller models are faster and cheaper to serve, so fine-tuning is one of the highest-leverage tools for hitting latency and cost targets. + +Visit the following resources to learn more: + +- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md b/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md index a25edfa3b..655ffbbed 100644 --- a/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md +++ b/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md @@ -1,3 +1,10 @@ # FlashAttention -FlashAttention is a series of hand-optimized CUDA kernels that compute attention with fewer reads and writes to GPU memory by fusing the attention algorithm into a single kernel. Standard attention stores intermediate matrices to memory between steps; FlashAttention eliminates those round-trips. FlashAttention 3 targets Hopper GPUs and FlashAttention 4 targets Blackwell GPUs. \ No newline at end of file +FlashAttention is a series of hand-optimized CUDA kernels that compute attention with fewer reads and writes to GPU memory by fusing the attention algorithm into a single kernel. Standard attention stores intermediate matrices to memory between steps; FlashAttention eliminates those round-trips. FlashAttention 3 targets Hopper GPUs and FlashAttention 4 targets Blackwell GPUs. + +Visit the following resources to learn more: + +- [@opensource@Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention) +- [@article@FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135) +- [@article@FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://arxiv.org/abs/2307.08691) +- [@article@FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://arxiv.org/abs/2407.08608) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md b/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md index d2489002e..1b22c166f 100644 --- a/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md +++ b/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md @@ -1,3 +1,9 @@ # GPU Architecture -GPU architecture determines the hard limits of what inference can achieve on a given device. This group covers compute, memory, interconnects, and sizing — the four dimensions an inference engineer needs to understand to make informed decisions about hardware selection, parallelism strategy, and optimization priorities. \ No newline at end of file +GPU architecture determines the hard limits of what inference can achieve on a given device. This group covers compute, memory, interconnects, and sizing — the four dimensions an inference engineer needs to understand to make informed decisions about hardware selection, parallelism strategy, and optimization priorities. + +Visit the following resources to learn more: + +- [@article@NVIDIA Tesla: A Unified Graphics and Computing Architecture](https://ieeexplore.ieee.org/document/4523358) +- [@article@GPU Glossary](https://modal.com/gpu-glossary) +- [@book@Programming Massively Parallel Processors](https://www.goodreads.com/work/editions/10244675) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md b/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md index 68e0c6f68..0112073f7 100644 --- a/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md +++ b/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md @@ -1,3 +1,7 @@ # Grace and Vera CPUs -NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink Chip-to-Chip at up to 900 GB/s, far higher than standard PCIe connections. This high-bandwidth CPU-GPU link makes offloading KV caches and LoRA weights to CPU memory practical, since data can be retrieved quickly enough to remain on the critical path. Grace pairs with Hopper and Blackwell; Vera pairs with the upcoming Rubin architecture. \ No newline at end of file +NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink Chip-to-Chip at up to 900 GB/s, far higher than standard PCIe connections. This high-bandwidth CPU-GPU link makes offloading KV caches and LoRA weights to CPU memory practical, since data can be retrieved quickly enough to remain on the critical path. Grace pairs with Hopper and Blackwell; Vera pairs with the upcoming Rubin architecture. + +Visit the following resources to learn more: + +- [@official@NVIDIA Grace CPU](https://www.nvidia.com/en-us/data-center/grace-cpu/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md b/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md index fca29252e..88ff4a4f2 100644 --- a/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md +++ b/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md @@ -1,3 +1,7 @@ # Hopper -The Hopper architecture, released in 2022 with the H100 and later the H200, introduced FP8 Tensor Cores offering twice the compute throughput of FP16 and asynchronous data transfer features used by FlashAttention 3. H100 and H200 GPUs remain among the most widely deployed inference accelerators due to their maturity, broad software support, and relative availability compared to newer generations. \ No newline at end of file +The Hopper architecture, released in 2022 with the H100 and later the H200, introduced FP8 Tensor Cores offering twice the compute throughput of FP16 and asynchronous data transfer features used by FlashAttention 3. H100 and H200 GPUs remain among the most widely deployed inference accelerators due to their maturity, broad software support, and relative availability compared to newer generations. + +Visit the following resources to learn more: + +- [@official@NVIDIA H100 Tensor Core GPU Architecture](https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md b/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md index f5ebf793e..3f2b22ff0 100644 --- a/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md +++ b/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md @@ -1,3 +1,10 @@ # Image Generation Architecture -Image generation models are pipelines of multiple components: a text encoder that converts the prompt into conditioning, a denoising model that iteratively refines a latent representation from noise into an image, and a variational autoencoder that converts the final latent to pixels. The denoising model is the largest and most compute-intensive component and runs up to 100 forward passes per image generation. \ No newline at end of file +Image generation models are pipelines of multiple components: a text encoder that converts the prompt into conditioning, a denoising model that iteratively refines a latent representation from noise into an image, and a variational autoencoder that converts the final latent to pixels. The denoising model is the largest and most compute-intensive component and runs up to 100 forward passes per image generation. + +Visit the following resources to learn more: + +- [@article@Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) +- [@article@High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) +- [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) +- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md b/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md index df9ed063d..18a6ee6c0 100644 --- a/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md +++ b/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md @@ -1,3 +1,8 @@ # Image Generation Inference -Production image generation inference uses SGLang Diffusion, TensorRT, or hand-optimized PyTorch. The most important optimization target is the attention kernel, where FlashAttention 3 or 4 outperforms the default on Hopper and Blackwell respectively. GEMM kernels for linear layers can be quantized to FP8 safely. Torch compilation with manual kernel plugins caches the compiled engine to avoid multi-minute recompilation on every cold start. \ No newline at end of file +Production image generation inference uses SGLang Diffusion, TensorRT, or hand-optimized PyTorch. The most important optimization target is the attention kernel, where FlashAttention 3 or 4 outperforms the default on Hopper and Blackwell respectively. GEMM kernels for linear layers can be quantized to FP8 safely. Torch compilation with manual kernel plugins caches the compiled engine to avoid multi-minute recompilation on every cold start. + +Visit the following resources to learn more: + +- [@opensource@ComfyUI](https://github.com/comfyanonymous/ComfyUI) +- [@article@Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md b/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md index 67d74abfe..7fc1c4c8b 100644 --- a/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md +++ b/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md @@ -1,3 +1,7 @@ # Inference vs. Training -Training is the process of learning model weights from data. Inference is serving those weights in production to generate outputs. Training runs are compute-intensive batch jobs; inference is a continuous, latency-sensitive service. Where training success is measured in loss curves, inference success is measured in time to first token, tokens per second, and uptime. \ No newline at end of file +Training is the process of learning model weights from data. Inference is serving those weights in production to generate outputs. Training runs are compute-intensive batch jobs; inference is a continuous, latency-sensitive service. Where training success is measured in loss curves, inference success is measured in time to first token, tokens per second, and uptime. + +Visit the following resources to learn more: + +- [@article@Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md b/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md index 6ae437c1a..4d70d52fe 100644 --- a/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md +++ b/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md @@ -1,3 +1,8 @@ # Instances and Interconnects -Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, and NVSwitch creates an all-to-all fabric across all eight GPUs. Between nodes, InfiniBand provides inter-node connectivity at substantially lower bandwidth than NVLink. Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated slices, each with dedicated compute and memory, allowing small models to run on fractions of a high-end GPU without resource contention. \ No newline at end of file +Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, and NVSwitch creates an all-to-all fabric across all eight GPUs. Between nodes, InfiniBand provides inter-node connectivity at substantially lower bandwidth than NVLink. Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated slices, each with dedicated compute and memory, allowing small models to run on fractions of a high-end GPU without resource contention. + +Visit the following resources to learn more: + +- [@official@NVLink and NVSwitch](https://www.nvidia.com/en-us/data-center/nvlink/) +- [@official@NVIDIA InfiniBand](https://www.nvidia.com/en-us/networking/products/infiniband/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md b/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md index 3472c860c..68c41d093 100644 --- a/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md +++ b/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md @@ -1,3 +1,8 @@ # KV Cache Storage -KV caches can be stored in GPU VRAM (fastest, tens to hundreds of gigabytes), CPU RAM (slower, hundreds of gigabytes to terabytes), local SSD, or networked SSD. Moving less frequently accessed KV blocks to slower storage frees VRAM for active requests. NVIDIA Dynamo's KV Block Manager provides APIs for managing KV blocks across these storage tiers transparently. \ No newline at end of file +KV caches can be stored in GPU VRAM (fastest, tens to hundreds of gigabytes), CPU RAM (slower, hundreds of gigabytes to terabytes), local SSD, or networked SSD. Moving less frequently accessed KV blocks to slower storage frees VRAM for active requests. NVIDIA Dynamo's KV Block Manager provides APIs for managing KV blocks across these storage tiers transparently. + +Visit the following resources to learn more: + +- [@opensource@LMCache](https://github.com/LMCache/LMCache) +- [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md b/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md index b921c7003..b2c02c6e6 100644 --- a/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md +++ b/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md @@ -1,3 +1,7 @@ # KV Cache -The KV cache stores the key and value tensors computed during attention for each previously processed token. Without it, attention would require recomputing every prior token on each decode step, making inference prohibitively slow. The KV cache lives in GPU memory and its size grows with sequence length and batch size, making it a major target for memory management techniques. \ No newline at end of file +The KV cache stores the key and value tensors computed during attention for each previously processed token. Without it, attention would require recomputing every prior token on each decode step, making inference prohibitively slow. The KV cache lives in GPU memory and its size grows with sequence length and batch size, making it a major target for memory management techniques. + +Visit the following resources to learn more: + +- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md index 4133a9a98..9092ef62d 100644 --- a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md +++ b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md @@ -1,3 +1,7 @@ # Large Language Models -An LLM is a decoder-only transformer with three main components: an embedding layer that converts input tokens to vectors, a stack of transformer blocks that process those vectors, and an output language modeling head that converts the final hidden states into a probability distribution over the vocabulary. The configuration file for an open model specifies all architectural details including layer count, hidden dimensions, and attention heads. \ No newline at end of file +An LLM is a decoder-only transformer with three main components: an embedding layer that converts input tokens to vectors, a stack of transformer blocks that process those vectors, and an output language modeling head that converts the final hidden states into a probability distribution over the vocabulary. The configuration file for an open model specifies all architectural details including layer count, hidden dimensions, and attention heads. + +Visit the following resources to learn more: + +- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md b/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md index 3eb73dde1..8975a26a8 100644 --- a/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md +++ b/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md @@ -1,3 +1,8 @@ # LLM Inference Bottlenecks -LLM prefill is compute-bound because it processes the entire input sequence as one large matrix multiplication, performing many operations per byte loaded. LLM decode is memory-bound because it generates tokens one at a time via vector-matrix multiplications that require loading all model weights from memory for each token while performing relatively few arithmetic operations. \ No newline at end of file +LLM prefill is compute-bound because it processes the entire input sequence as one large matrix multiplication, performing many operations per byte loaded. LLM decode is memory-bound because it generates tokens one at a time via vector-matrix multiplications that require loading all model weights from memory for each token while performing relatively few arithmetic operations. + +Visit the following resources to learn more: + +- [@article@GPU Glossary](https://modal.com/gpu-glossary) +- [@book@AI Systems Performance Engineering](https://www.oreilly.com/library/view/ai-systems-performance/9781098170202/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md b/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md index 4d827c5f6..dc5aba8c7 100644 --- a/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md +++ b/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md @@ -1,3 +1,8 @@ # Long Context Handling -Long sequences produce KV caches large enough to strain GPU memory, slow down decode, and increase prefill time. Techniques for handling long context include FlashAttention for memory-efficient attention computation, PagedAttention for non-contiguous KV storage, chunked prefill to interleave long prefill with decode, and model-level approaches like sliding window attention that limit how many prior tokens each token attends to. \ No newline at end of file +Long sequences produce KV caches large enough to strain GPU memory, slow down decode, and increase prefill time. Techniques for handling long context include FlashAttention for memory-efficient attention computation, PagedAttention for non-contiguous KV storage, chunked prefill to interleave long prefill with decode, and model-level approaches like sliding window attention that limit how many prior tokens each token attends to. + +Visit the following resources to learn more: + +- [@article@Ring Attention with Blockwise Transformers for Near-Infinite Context](https://arxiv.org/abs/2310.01889) +- [@article@RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md b/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md index 5062fa767..3464eef3d 100644 --- a/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md +++ b/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md @@ -1,3 +1,7 @@ # Mixture of Experts (MoE) -Mixture of Experts models replace dense linear layers with a set of smaller expert matrices, activating only a subset of experts per token via a learned router. A model like Qwen3-235B activates only 22 billion of its 235 billion parameters per forward pass, making single-user inference efficient. In batched production serving, different tokens activate different experts, so nearly all parameters are active across the batch. \ No newline at end of file +Mixture of Experts models replace dense linear layers with a set of smaller expert matrices, activating only a subset of experts per token via a learned router. A model like Qwen3-235B activates only 22 billion of its 235 billion parameters per forward pass, making single-user inference efficient. In batched production serving, different tokens activate different experts, so nearly all parameters are active across the batch. + +Visit the following resources to learn more: + +- [@article@Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md b/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md index 37ab15cca..9fbd6f1b5 100644 --- a/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md +++ b/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md @@ -1,3 +1,8 @@ # Model Evaluation -Model evaluation, or evals, is the practice of systematically measuring model quality for a specific task or domain. Standard intelligence benchmarks like MMLU measure general capabilities but can be gamed; product-specific evals that test real use cases provide more reliable signal. High-confidence evals are a prerequisite before investing in inference optimization, ensuring the model is actually useful before making it fast. \ No newline at end of file +Model evaluation, or evals, is the practice of systematically measuring model quality for a specific task or domain. Standard intelligence benchmarks like MMLU measure general capabilities but can be gamed; product-specific evals that test real use cases provide more reliable signal. High-confidence evals are a prerequisite before investing in inference optimization, ensuring the model is actually useful before making it fast. + +Visit the following resources to learn more: + +- [@article@How to Fine-Tune Qwen3 to GPT-4o Level Performance](https://ghost.oxen.ai/how-to-fine-tune-qwen3-to-gpt-4o-level-performance/) +- [@book@Evals for AI Engineers](https://www.oreilly.com/library/view/evals-for-ai/9798341660717/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md b/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md index ec0140381..4d05c2e9a 100644 --- a/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md +++ b/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md @@ -1,3 +1,7 @@ # Model Selection -The most important decision in model performance optimization is which model to use. Smaller models with fewer parameters are faster and cheaper to serve than larger ones. The goal is to find the smallest model that meets quality requirements for the task. Model architecture also determines which inference engines and optimization techniques are available, so popular architectures with broad tooling support are preferable. \ No newline at end of file +The most important decision in model performance optimization is which model to use. Smaller models with fewer parameters are faster and cheaper to serve than larger ones. The goal is to find the smallest model that meets quality requirements for the task. Model architecture also determines which inference engines and optimization techniques are available, so popular architectures with broad tooling support are preferable. + +Visit the following resources to learn more: + +- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md b/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md index 497240a0c..0f678708d 100644 --- a/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md +++ b/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md @@ -1,3 +1,10 @@ -# Multi-Cloud +# Multi-Cloud -Multi-cloud inference distributes GPU capacity across providers and regions to maximize availability, minimize latency, and meet data residency requirements. This group covers procurement strategy, capacity management, geographic routing, and the reliability patterns that make large-scale inference resilient to hardware and provider failures. \ No newline at end of file +Multi-cloud inference distributes GPU capacity across providers and regions to maximize availability, minimize latency, and meet data residency requirements. This group covers procurement strategy, capacity management, geographic routing, and the reliability patterns that make large-scale inference resilient to hardware and provider failures. + +Visit the following resources to learn more: + +- [@article@SemiAnalysis](https://semianalysis.com/) +- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) +- [@book@Designing Data-Intensive Applications](https://dataintensive.net/) +- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md b/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md index c2661df62..3dbc4580e 100644 --- a/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md +++ b/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md @@ -1,3 +1,7 @@ # N-gram Speculation -N-gram speculation builds a dictionary during prefill that maps token prefixes to observed n-gram continuations from the input. During decode, generated tokens are looked up in this dictionary and matching suffixes become draft tokens with no separate model required. It is most effective for code completion, where output frequently mirrors input syntax, and can produce very long draft sequences with high acceptance rates in that domain. \ No newline at end of file +N-gram speculation builds a dictionary during prefill that maps token prefixes to observed n-gram continuations from the input. During decode, generated tokens are looked up in this dictionary and matching suffixes become draft tokens with no separate model required. It is most effective for code completion, where output frequently mirrors input syntax, and can produce very long draft sequences with high acceptance rates in that domain. + +Visit the following resources to learn more: + +- [@article@Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://arxiv.org/abs/2402.02057) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md b/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md index bcf41deed..1df6cd1e0 100644 --- a/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md +++ b/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md @@ -1,3 +1,9 @@ # Neural Networks -A neural network is a layered structure of nodes where each node applies a learned weight matrix and bias to its input. Multiple layers connected in sequence form the foundation for all generative AI models. Inference engineers need a working model of how neural networks compute outputs to understand where performance bottlenecks arise. \ No newline at end of file +A neural network is a layered structure of nodes where each node applies a learned weight matrix and bias to its input. Multiple layers connected in sequence form the foundation for all generative AI models. Inference engineers need a working model of how neural networks compute outputs to understand where performance bottlenecks arise. + +Visit the following resources to learn more: + +- [@book@Deep Learning](https://www.deeplearningbook.org/) +- [@book@Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python-second-edition) +- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md b/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md index 75dcbc6e8..b873fe678 100644 --- a/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md +++ b/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md @@ -1,3 +1,7 @@ # NVIDIA Dynamo -Dynamo supports conditional disaggregation, where requests are first sent to the decode engine and only routed to a prefill engine when the input is long or not already cached. The ratio of prefill to decode engines is configurable at runtime and can shift as traffic patterns change. This dynamic allocation prevents the prefill queue from becoming a bottleneck and avoids wasting decode engine resources on prefill-heavy traffic. \ No newline at end of file +Dynamo supports conditional disaggregation, where requests are first sent to the decode engine and only routed to a prefill engine when the input is long or not already cached. The ratio of prefill to decode engines is configurable at runtime and can shift as traffic patterns change. This dynamic allocation prevents the prefill queue from becoming a bottleneck and avoids wasting decode engine resources on prefill-heavy traffic. + +Visit the following resources to learn more: + +- [@official@NVIDIA Dynamo Documentation](https://docs.nvidia.com/dynamo/latest/index.html) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md b/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md index ec30a5544..7d5dccca1 100644 --- a/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md +++ b/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md @@ -1,3 +1,7 @@ # NVIDIA Dynamo -Dynamo is NVIDIA's open-source distributed serving platform that sits above inference engines and provides orchestration for large-scale deployments. It adds KV-aware routing, disaggregated prefill/decode serving, multi-node parallelism, and an SLA-based planner that dynamically adjusts the number of prefill and decode workers based on TTFT and TPS targets. \ No newline at end of file +Dynamo is NVIDIA's open-source distributed serving platform that sits above inference engines and provides orchestration for large-scale deployments. It adds KV-aware routing, disaggregated prefill/decode serving, multi-node parallelism, and an SLA-based planner that dynamically adjusts the number of prefill and decode workers based on TTFT and TPS targets. + +Visit the following resources to learn more: + +- [@official@NVIDIA Dynamo Documentation](https://docs.nvidia.com/dynamo/latest/index.html) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md b/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md index f836629ba..b66f800cb 100644 --- a/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md +++ b/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md @@ -1,3 +1,7 @@ # Observability -Inference observability tracks total request volume, input and output sequence lengths, response codes, latency percentiles (P50, P90, P99 of TTFT, TPS, and end-to-end latency), replica count, GPU utilization, and queue depth. These metrics are interdependent; a latency spike could come from long input sequences, high concurrency, or a hardware issue. Observability must integrate with existing tooling like Grafana, Datadog, and PagerDuty to put inference metrics in context. \ No newline at end of file +Inference observability tracks total request volume, input and output sequence lengths, response codes, latency percentiles (P50, P90, P99 of TTFT, TPS, and end-to-end latency), replica count, GPU utilization, and queue depth. These metrics are interdependent; a latency spike could come from long input sequences, high concurrency, or a hardware issue. Observability must integrate with existing tooling like Grafana, Datadog, and PagerDuty to put inference metrics in context. + +Visit the following resources to learn more: + +- [@official@NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md b/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md index 931ee23f2..8c409ae95 100644 --- a/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md +++ b/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md @@ -1,3 +1,7 @@ # ONNX Runtime -ONNX Runtime is an open-source inference runtime that executes models exported to the ONNX format, which stores both weights and a computation graph for cross-platform portability. It supports multiple hardware backends including CUDA, DirectML, and CPU. It is widely used for embedding models, vision models, and pipelines that need to run across hardware environments without engine-specific configuration. \ No newline at end of file +ONNX Runtime is an open-source inference runtime that executes models exported to the ONNX format, which stores both weights and a computation graph for cross-platform portability. It supports multiple hardware backends including CUDA, DirectML, and CPU. It is widely used for embedding models, vision models, and pipelines that need to run across hardware environments without engine-specific configuration. + +Visit the following resources to learn more: + +- [@official@ONNX Runtime](https://onnxruntime.ai/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md b/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md index 3fc3fea0d..0c7606d17 100644 --- a/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md +++ b/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md @@ -1,3 +1,7 @@ # Open vs. Closed Models -Closed models are accessed exclusively through a vendor API; the weights are never released. Open models publish their weights for download and self-hosting. For inference engineers, this distinction determines the entire problem space: closed models require no infrastructure but offer no control over latency, cost, or configuration, while open models require building and operating a full inference stack but give complete control over every layer of it. \ No newline at end of file +Closed models are accessed exclusively through a vendor API; the weights are never released. Open models publish their weights for download and self-hosting. For inference engineers, this distinction determines the entire problem space: closed models require no infrastructure but offer no control over latency, cost, or configuration, while open models require building and operating a full inference stack but give complete control over every layer of it. + +Visit the following resources to learn more: + +- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md b/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md index cb06cce3e..e95d3254c 100644 --- a/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md +++ b/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md @@ -1,3 +1,7 @@ # PagedAttention -PagedAttention stores the KV cache in fixed-size pages rather than a single contiguous memory block, allowing it to be distributed across GPU memory without fragmentation. It uses a lookup table to access pages on demand, reducing memory waste from over-allocation and enabling more requests to share GPU memory. It is the memory management technique that made vLLM's high throughput possible. \ No newline at end of file +PagedAttention stores the KV cache in fixed-size pages rather than a single contiguous memory block, allowing it to be distributed across GPU memory without fragmentation. It uses a lookup table to access pages on demand, reducing memory waste from over-allocation and enabling more requests to share GPU memory. It is the memory management technique that made vLLM's high throughput possible. + +Visit the following resources to learn more: + +- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md b/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md index d2534a0d2..7ca6319aa 100644 --- a/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md +++ b/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md @@ -1,3 +1,8 @@ # Pipeline Parallelism -Pipeline Parallelism splits an LLM's layers into sequential stages and assigns each stage to a different GPU or node. Unlike Tensor Parallelism, each GPU holds complete layers rather than layer fragments. It introduces pipeline bubbles where GPUs wait for preceding stages, reducing efficiency. It is used for multi-node inference with dense models, typically combined with Tensor Parallelism within each node. \ No newline at end of file +Pipeline Parallelism splits an LLM's layers into sequential stages and assigns each stage to a different GPU or node. Unlike Tensor Parallelism, each GPU holds complete layers rather than layer fragments. It introduces pipeline bubbles where GPUs wait for preceding stages, reducing efficiency. It is used for multi-node inference with dense models, typically combined with Tensor Parallelism within each node. + +Visit the following resources to learn more: + +- [@opensource@NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM) +- [@article@Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/post-training-quantization@Zvvs8cB98IRee1okBILsS.md b/roadmaps/inference-engineering/content/post-training-quantization@Zvvs8cB98IRee1okBILsS.md index 639431c98..682c6c536 100644 --- a/roadmaps/inference-engineering/content/post-training-quantization@Zvvs8cB98IRee1okBILsS.md +++ b/roadmaps/inference-engineering/content/post-training-quantization@Zvvs8cB98IRee1okBILsS.md @@ -1,3 +1,12 @@ -Post-Training Quantization +# Post-Training Quantization -Post-training quantization converts finished model weights to lower precision after training is complete, using a small calibration dataset to compute per-layer scale factors that minimize the gap between quantized and original outputs. It is the standard approach for most deployments since it requires no retraining and can be applied to any existing model. The leading tool is NVIDIA ModelOpt, which exports quantized models in formats compatible with all major inference engines. \ No newline at end of file +Post-training quantization converts finished model weights to lower precision after training is complete, using a small calibration dataset to compute per-layer scale factors that minimize the gap between quantized and original outputs. It is the standard approach for most deployments since it requires no retraining and can be applied to any existing model. The leading tool is NVIDIA ModelOpt, which exports quantized models in formats compatible with all major inference engines. + +Visit the following resources to learn more: + +- [@opensource@AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://github.com/mit-han-lab/llm-awq) +- [@opensource@bitsandbytes](https://github.com/TimDettmers/bitsandbytes) +- [@opensource@SmoothQuant](https://github.com/mit-han-lab/smoothquant) +- [@article@GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) +- [@article@LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339) +- [@article@SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/abs/2301.00774) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md b/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md index 120c93c12..efc6f483c 100644 --- a/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md +++ b/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md @@ -1,3 +1,8 @@ # Profiling Performance -Profiling reveals where time is spent within a benchmark, identifying which operations consume the most compute or memory. Tools include PyTorch Profiler, NVIDIA Nsight Systems for system-wide GPU and CPU analysis, and NVIDIA Nsight Compute for per-kernel analysis. Profiling is most useful when writing custom inference code or contributing to inference frameworks, rather than configuring existing engines. \ No newline at end of file +Profiling reveals where time is spent within a benchmark, identifying which operations consume the most compute or memory. Tools include PyTorch Profiler, NVIDIA Nsight Systems for system-wide GPU and CPU analysis, and NVIDIA Nsight Compute for per-kernel analysis. Profiling is most useful when writing custom inference code or contributing to inference frameworks, rather than configuring existing engines. + +Visit the following resources to learn more: + +- [@official@PyTorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) +- [@official@NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md b/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md index 753411aeb..0bbb52b09 100644 --- a/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md +++ b/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md @@ -1,3 +1,7 @@ # PyTorch -PyTorch is the standard Python framework for building and running neural networks. It abstracts CUDA into a tensor operation API and provides automatic differentiation for training. For inference, PyTorch supports torch.compile for automatic kernel optimization and serves as the foundation that all major inference engines build on top of. \ No newline at end of file +PyTorch is the standard Python framework for building and running neural networks. It abstracts CUDA into a tensor operation API and provides automatic differentiation for training. For inference, PyTorch supports torch.compile for automatic kernel optimization and serves as the foundation that all major inference engines build on top of. + +Visit the following resources to learn more: + +- [@official@PyTorch Performance Tuning Guide](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md b/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md index 62e4bf0e9..264447515 100644 --- a/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md +++ b/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md @@ -1,3 +1,12 @@ # Quantization -Quantization converts model weights and other values from their native precision to a lower-precision format. Cutting precision in half doubles memory bandwidth for decode and enables twice the FLOPS for prefill on compatible Tensor Cores. The tradeoff is potential quality loss from reduced numerical precision, which can compound across the many calculations in a forward pass. \ No newline at end of file +Quantization converts model weights and other values from their native precision to a lower-precision format. Cutting precision in half doubles memory bandwidth for decode and enables twice the FLOPS for prefill on compatible Tensor Cores. The tradeoff is potential quality loss from reduced numerical precision, which can compound across the many calculations in a forward pass. + +Visit the following resources to learn more: + +- [@opensource@AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://github.com/mit-han-lab/llm-awq) +- [@opensource@bitsandbytes](https://github.com/TimDettmers/bitsandbytes) +- [@opensource@SmoothQuant](https://github.com/mit-han-lab/smoothquant) +- [@article@GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) +- [@article@LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339) +- [@article@SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/abs/2301.00774) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md b/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md index 36bb07050..9acda65b9 100644 --- a/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md +++ b/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md @@ -1,3 +1,9 @@ # Routing and Load Balancing -Routers make request-level decisions about which replica to send a request to, considering factors like KV cache prefix match and LoRA availability. Load balancers make system-level decisions to distribute traffic evenly across replicas. In systems with prefix caching, naive round-robin load balancing misses cache opportunities; intelligent routing increases cache hit rates and reduces TTFT. \ No newline at end of file +Routers make request-level decisions about which replica to send a request to, considering factors like KV cache prefix match and LoRA availability. Load balancers make system-level decisions to distribute traffic evenly across replicas. In systems with prefix caching, naive round-robin load balancing misses cache opportunities; intelligent routing increases cache hit rates and reduces TTFT. + +Visit the following resources to learn more: + +- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) +- [@book@Designing Data-Intensive Applications](https://dataintensive.net/) +- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md b/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md index 9b16777c7..2bb16ac62 100644 --- a/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md +++ b/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md @@ -1,3 +1,7 @@ # SGLang -SGLang is a high-throughput inference engine with a flexible frontend language for customizing server behavior. It has strong support for MoE models like DeepSeek and Qwen and works closely with model labs to optimize new architectural features. SGLang Diffusion extends it to image and video generation models with pipeline abstractions and diffusion-specific parallelism. \ No newline at end of file +SGLang is a high-throughput inference engine with a flexible frontend language for customizing server behavior. It has strong support for MoE models like DeepSeek and Qwen and works closely with model labs to optimize new architectural features. SGLang Diffusion extends it to image and video generation models with pipeline abstractions and diffusion-specific parallelism. + +Visit the following resources to learn more: + +- [@opensource@sgl-project/sglang](https://github.com/sgl-project/sglang) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md b/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md index 96efcfae6..74e7213cd 100644 --- a/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md +++ b/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md @@ -1,3 +1,9 @@ # Sizing and Procurement -Estimating GPU requirements starts with VRAM: roughly 1 GB per billion parameters at FP8 for weights, plus at least 50 percent headroom for the KV cache. This total is rounded up to the nearest available instance size. Cloud GPUs are available from hyperscalers like AWS and GCP, neoclouds like CoreWeave and Nebius, and spot markets; large deployments combine reserved instances for baseline load with on-demand and spot capacity for bursts. \ No newline at end of file +Estimating GPU requirements starts with VRAM: roughly 1 GB per billion parameters at FP8 for weights, plus at least 50 percent headroom for the KV cache. This total is rounded up to the nearest available instance size. Cloud GPUs are available from hyperscalers like AWS and GCP, neoclouds like CoreWeave and Nebius, and spot markets; large deployments combine reserved instances for baseline load with on-demand and spot capacity for bursts. + +Visit the following resources to learn more: + +- [@article@SemiAnalysis](https://semianalysis.com/) +- [@book@Designing Data-Intensive Applications](https://dataintensive.net/) +- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md b/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md index dd768cce6..0e185c989 100644 --- a/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md +++ b/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md @@ -1,3 +1,8 @@ # Speculative Decoding -Speculative decoding takes advantage of spare compute during memory-bound decode by generating multiple draft tokens and validating them in a single forward pass. If the target model accepts N draft tokens, the forward pass produces N+1 tokens instead of one. The performance gain depends on draft token cost, draft sequence length, and acceptance rate, which decreases at higher temperatures and larger batch sizes. \ No newline at end of file +Speculative decoding takes advantage of spare compute during memory-bound decode by generating multiple draft tokens and validating them in a single forward pass. If the target model accepts N draft tokens, the forward pass produces N+1 tokens instead of one. The performance gain depends on draft token cost, draft sequence length, and acceptance rate, which decreases at higher temperatures and larger batch sizes. + +Visit the following resources to learn more: + +- [@article@Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) +- [@article@Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://arxiv.org/abs/2401.10774) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md b/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md index 07d8e654e..01e480138 100644 --- a/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md +++ b/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md @@ -1,3 +1,8 @@ # Tensor Parallelism -Tensor Parallelism splits each layer's weight matrices across multiple GPUs and distributes the matmul computation. Each GPU handles a shard of each layer and the results are combined via an all-reduce operation before the next layer. It reduces per-user latency by sharing both VRAM and compute across GPUs, but requires fast NVLink interconnects and is not well-suited for multi-node deployments. \ No newline at end of file +Tensor Parallelism splits each layer's weight matrices across multiple GPUs and distributes the matmul computation. Each GPU handles a shard of each layer and the results are combined via an all-reduce operation before the next layer. It reduces per-user latency by sharing both VRAM and compute across GPUs, but requires fast NVLink interconnects and is not well-suited for multi-node deployments. + +Visit the following resources to learn more: + +- [@opensource@NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM) +- [@article@Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md b/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md index 9d418284c..88c0bba93 100644 --- a/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md +++ b/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md @@ -1,3 +1,7 @@ # TensorRT-LLM -TensorRT-LLM is NVIDIA's inference engine combining PyTorch-based model execution with proprietary NVIDIA kernels unavailable in other engines. It achieves the highest performance among the three major engines, with excellent support for Hopper and Blackwell architectures and NVIDIA-specific formats like NVFP4. The tradeoff is a steeper learning curve and narrower model support compared to vLLM or SGLang. \ No newline at end of file +TensorRT-LLM is NVIDIA's inference engine combining PyTorch-based model execution with proprietary NVIDIA kernels unavailable in other engines. It achieves the highest performance among the three major engines, with excellent support for Hopper and Blackwell architectures and NVIDIA-specific formats like NVFP4. The tradeoff is a steeper learning curve and narrower model support compared to vLLM or SGLang. + +Visit the following resources to learn more: + +- [@opensource@NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md b/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md index 9d0e866ea..f8dcc59d4 100644 --- a/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md +++ b/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md @@ -1,3 +1,7 @@ # TensorRT -TensorRT is NVIDIA's high-performance inference runtime that compiles PyTorch or ONNX models into optimized engines using fused kernels, quantization, and hardware-specific tuning. It is the standard choice for image and video generation models and for any workload where maximum throughput on NVIDIA hardware is the primary goal. Compiled engines are cached to disk, eliminating recompilation overhead on subsequent cold starts. \ No newline at end of file +TensorRT is NVIDIA's high-performance inference runtime that compiles PyTorch or ONNX models into optimized engines using fused kernels, quantization, and hardware-specific tuning. It is the standard choice for image and video generation models and for any workload where maximum throughput on NVIDIA hardware is the primary goal. Compiled engines are cached to disk, eliminating recompilation overhead on subsequent cold starts. + +Visit the following resources to learn more: + +- [@official@TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md b/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md index 98dc09529..097af156b 100644 --- a/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md +++ b/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md @@ -1,3 +1,7 @@ # Transformer Blocks -Transformer blocks are the repeated hidden layers of an LLM. Each block contains an attention sublayer, a feed-forward neural network sublayer, and normalization layers. The feed-forward network holds the majority of model parameters, while attention is the most computationally complex operation per token. \ No newline at end of file +Transformer blocks are the repeated hidden layers of an LLM. Each block contains an attention sublayer, a feed-forward neural network sublayer, and normalization layers. The feed-forward network holds the majority of model parameters, while attention is the most computationally complex operation per token. + +Visit the following resources to learn more: + +- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md b/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md index cde7c03ab..56e1f2bb5 100644 --- a/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md +++ b/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md @@ -1,3 +1,8 @@ # Hugging Face Transformers and Diffusers -The transformers and diffusers libraries provide reference implementations of popular LLM and image/video generation architectures. They are not designed for production serving but are essential for understanding model configurations, downloading weights, and prototyping inference. Model cards on Hugging Face use transformers or diffusers sample code as the canonical usage examples. \ No newline at end of file +The transformers and diffusers libraries provide reference implementations of popular LLM and image/video generation architectures. They are not designed for production serving but are essential for understanding model configurations, downloading weights, and prototyping inference. Model cards on Hugging Face use transformers or diffusers sample code as the canonical usage examples. + +Visit the following resources to learn more: + +- [@official@Hugging Face Diffusers](https://huggingface.co/docs/diffusers/index) +- [@official@Hugging Face Transformers](https://huggingface.co/docs/transformers/index) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md b/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md index d0427fbd3..e6b6bda00 100644 --- a/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md +++ b/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md @@ -1,3 +1,9 @@ # Video Generation Inference -Video generation models run on full nodes of eight GPUs with a batch size of one because the latent space is too large to batch effectively. Attention over the three-dimensional video latent space consumes 70 to 80 percent of compute time and is the primary optimization target. Attention caching techniques that reuse outputs from previous timesteps or transformer layers can improve speed by 30 to 40 percent. \ No newline at end of file +Video generation models run on full nodes of eight GPUs with a batch size of one because the latent space is too large to batch effectively. Attention over the three-dimensional video latent space consumes 70 to 80 percent of compute time and is the primary optimization target. Attention caching techniques that reuse outputs from previous timesteps or transformer layers can improve speed by 30 to 40 percent. + +Visit the following resources to learn more: + +- [@opensource@Cache-DiT](https://github.com/vipshop/cache-dit) +- [@opensource@TeaCache: Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://github.com/ali-vilab/TeaCache) +- [@article@SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://arxiv.org/abs/2410.02367) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md b/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md index 6698c248d..bb589aa69 100644 --- a/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md +++ b/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md @@ -1,3 +1,8 @@ # Video Generation Models -Video generation models extend image generation to three-dimensional latent space, representing width, height, and time simultaneously. They run the same number of denoising steps as image models but process far more data per step. Attention over the full video latent space dominates compute time, making video generation the most resource-intensive common inference workload. \ No newline at end of file +Video generation models extend image generation to three-dimensional latent space, representing width, height, and time simultaneously. They run the same number of denoising steps as image models but process far more data per step. Attention over the full video latent space dominates compute time, making video generation the most resource-intensive common inference workload. + +Visit the following resources to learn more: + +- [@article@Video Diffusion Models](https://arxiv.org/abs/2204.03458) +- [@article@Imagen Video: High Definition Video Generation with Diffusion Models](https://arxiv.org/abs/2210.02303) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md b/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md index 704121da9..e0c34469a 100644 --- a/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md +++ b/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md @@ -1,3 +1,10 @@ # Vision Language Models (VLMs) -VLMs combine a large language model with a small vision encoder that converts input images or video into a sequence of visual tokens. A high-resolution image typically adds about 1,000 tokens to the input sequence. All LLM inference optimization techniques apply to VLMs, but longer sequences make quantization, prefix caching, and disaggregation especially important. \ No newline at end of file +VLMs combine a large language model with a small vision encoder that converts input images or video into a sequence of visual tokens. A high-resolution image typically adds about 1,000 tokens to the input sequence. All LLM inference optimization techniques apply to VLMs, but longer sequences make quantization, prefix caching, and disaggregation especially important. + +Visit the following resources to learn more: + +- [@article@BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597) +- [@article@Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) +- [@article@Segment Anything](https://arxiv.org/abs/2304.02643) +- [@article@SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://arxiv.org/abs/2509.11815) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md b/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md index dd2d4680c..934a0d996 100644 --- a/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md +++ b/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md @@ -1,3 +1,7 @@ # vLLM -vLLM is the most widely adopted LLM inference engine, offering broad support for model architectures, hardware platforms (NVIDIA, AMD, Google TPUs), and optimization techniques including prefix caching, quantization, speculative decoding, and parallelism. Its primary advantage is ease of adoption and the fastest day-zero support for newly released open models. It also supports multimodal inference via vLLM Omni. \ No newline at end of file +vLLM is the most widely adopted LLM inference engine, offering broad support for model architectures, hardware platforms (NVIDIA, AMD, Google TPUs), and optimization techniques including prefix caching, quantization, speculative decoding, and parallelism. Its primary advantage is ease of adoption and the fastest day-zero support for newly released open models. It also supports multimodal inference via vLLM Omni. + +Visit the following resources to learn more: + +- [@opensource@vllm-project/vllm](https://github.com/vllm-project/vllm) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md b/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md index 2901724f5..1a3694d14 100644 --- a/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md +++ b/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md @@ -1,3 +1,12 @@ # Writing Custom Kernels -Custom kernels are written when existing libraries do not fully utilize a GPU's capabilities for a specific operation or architecture. Most production kernels are written in C++ using CUTLASS and CuTe for GEMM-style operations and FlashInfer for attention variants; Triton offers a Python-based alternative with a lower barrier to entry. Writing an effective kernel requires understanding the target GPU's memory hierarchy, SM occupancy, and Tensor Core instruction set — skills that sit at the intersection of software and hardware. \ No newline at end of file +Custom kernels are written when existing libraries do not fully utilize a GPU's capabilities for a specific operation or architecture. Most production kernels are written in C++ using CUTLASS and CuTe for GEMM-style operations and FlashInfer for attention variants; Triton offers a Python-based alternative with a lower barrier to entry. Writing an effective kernel requires understanding the target GPU's memory hierarchy, SM occupancy, and Tensor Core instruction set — skills that sit at the intersection of software and hardware. + +Visit the following resources to learn more: + +- [@official@CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/) +- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/) +- [@opensource@NVIDIA/cutlass](https://github.com/NVIDIA/cutlass) +- [@opensource@deepseek-ai/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM) +- [@article@FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://arxiv.org/abs/2501.01005) +- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example) \ No newline at end of file