From a261c81b0eddbcf344d210ea11ebf5d69e0c478b Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 2 Oct 2026 11:00:46 +0200 Subject: [PATCH] chore: sync content to repo (#10413) Co-authored-by: nilbuild <4921183+nilbuild@users.noreply.github.com> --- .../arithmetic-intensity@bPO6uwod30-ewwbo8i1eU.md | 9 ++++++++- .../content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md | 5 ++++- .../attention-optimization@KWM3V7_15_Aur0He9j2Va.md | 8 +++++++- .../content/blackwell@LihLABERuKBw30G_j48Fj.md | 4 +++- .../content/bottleneck-analysis@Gtd4g4MtB9zFfahRpbE2K.md | 9 ++++++++- .../content/compute@o_tknzA4tqcpbU6t5HO-T.md | 9 ++++++++- .../content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md | 3 ++- .../content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md | 7 ++----- .../encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md | 2 +- .../few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md | 3 ++- .../content/flashattention@mQcj-H3R789o6nDh8tG9k.md | 4 ++-- .../content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md | 5 +++-- .../content/gpu-models@umz3jScWQViO9-lACUU4f.md | 8 +++++++- .../content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md | 5 ++++- .../content/hopper@XQMbaW1tFdO8Os6mxcR2M.md | 5 ++++- .../image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md | 9 ++++++++- .../content/image--video-models@Mebx7HS-I-eNP5i3f6Clf.md | 9 ++++++++- .../image-generation-arch@xsV4WevPzhfydyvxVzGcw.md | 6 ++---- .../instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md | 4 +++- .../content/kv-cache@V678kX8FPChTZCe7Nqwhv.md | 5 ++++- .../large-language-models@irgGvEnY1_WW1EPtuMKgC.md | 2 +- .../content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md | 4 +++- .../local--edge-inference@FilQGz3geyyvHZGyKPW3G.md | 9 ++++++++- .../content/memory@5829d5dvXq9qlMSD1CG0N.md | 9 ++++++++- .../mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md | 5 ++++- .../content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md | 8 +++++++- .../content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md | 7 ++++++- .../content/opsbyte-ratio@uc2enkdtRqA5K7kbgrYJF.md | 7 ++++++- .../content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md | 8 +++++++- .../content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md | 4 +++- .../content/roofline-model@XF-wMQCqPZiWv1XmryPBP.md | 8 +++++++- .../content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md | 4 +++- .../content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md | 5 ++++- .../video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md | 4 +++- 34 files changed, 160 insertions(+), 43 deletions(-) diff --git a/roadmaps/inference-engineering/content/arithmetic-intensity@bPO6uwod30-ewwbo8i1eU.md b/roadmaps/inference-engineering/content/arithmetic-intensity@bPO6uwod30-ewwbo8i1eU.md index f51a86c1b..e9028fa47 100644 --- a/roadmaps/inference-engineering/content/arithmetic-intensity@bPO6uwod30-ewwbo8i1eU.md +++ b/roadmaps/inference-engineering/content/arithmetic-intensity@bPO6uwod30-ewwbo8i1eU.md @@ -1,3 +1,10 @@ # Arithmetic Intensity -Arithmetic intensity is a property of an algorithm: the number of floating-point operations performed per byte of memory read or written. A large matrix multiplication has high arithmetic intensity because the same weights participate in many operations per load; a decode forward pass has low arithmetic intensity because each weight is loaded from VRAM and used once per token. Comparing a workload's arithmetic intensity against the GPU's ops:byte ratio is the fastest way to diagnose whether adding compute or adding bandwidth would help. \ No newline at end of file +Arithmetic intensity is a property of an algorithm: the number of floating-point operations performed per byte of memory read or written. A large matrix multiplication has high arithmetic intensity because the same weights participate in many operations per load; a decode forward pass has low arithmetic intensity because each weight is loaded from VRAM and used once per token. Comparing a workload's arithmetic intensity against the GPU's ops:byte ratio is the fastest way to diagnose whether adding compute or adding bandwidth would help. + +Visit the following resources to learn more: + +- [@article@A guide to LLM inference and performance](https://www.baseten.co/blog/llm-transformer-inference-guide/) +- [@article@Arithmetic Intensity : Understand Op limits — Memory or Compute](https://medium.com/better-ml/arithmetic-intensity-understand-op-limits-memory-or-compute-342fd15342bb) +- [@article@Arithmetic Intensity Explained: The Formula and Why It Predicts GPU Speedup](https://ayoob.ai/blog/webgpu-matrix-multiplication-gemm-arithmetic-intensity) +- [@video@Compute-Bound or Memory-Bound? Arithmetic Intensity Explained](https://www.youtube.com/watch?v=jCUWDXVohTY) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md b/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md index 1aeab91ff..897658b6b 100644 --- a/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md +++ b/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md @@ -4,4 +4,7 @@ Attention is the mechanism transformers use to relate each token to all previous Visit the following resources to learn more: -- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762) \ No newline at end of file +- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762) +- [@article@What is an attention mechanism?](https://www.ibm.com/think/topics/attention-mechanism) +- [@video@Attention mechanism: Overview](https://www.youtube.com/watch?v=fjJOgb-E41w&t=1s) +- [@video@Attention in transformers, step-by-step](https://www.youtube.com/watch?v=eMlx5fFNoYc) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/attention-optimization@KWM3V7_15_Aur0He9j2Va.md b/roadmaps/inference-engineering/content/attention-optimization@KWM3V7_15_Aur0He9j2Va.md index a3b6354aa..c78b0a291 100644 --- a/roadmaps/inference-engineering/content/attention-optimization@KWM3V7_15_Aur0He9j2Va.md +++ b/roadmaps/inference-engineering/content/attention-optimization@KWM3V7_15_Aur0He9j2Va.md @@ -1,3 +1,9 @@ # Attention Optimization -Attention optimization refers to techniques designed to improve the computational efficiency and memory usage of the attention mechanism within transformer models. Because standard attention scales quadratically with input sequence length, these optimizations aim to reduce computation and memory overhead. By streamlining how tokens interact with one another, these methods allow models to process longer inputs faster and run on hardware with less available memory. \ No newline at end of file +Attention optimization refers to techniques designed to improve the computational efficiency and memory usage of the attention mechanism within transformer models. Because standard attention scales quadratically with input sequence length, these optimizations aim to reduce computation and memory overhead. By streamlining how tokens interact with one another, these methods allow models to process longer inputs faster and run on hardware with less available memory. + +Visit the following resources to learn more: + +- [@article@Attention Mechanism Innovation: Next-Generation Focus](https://translated.com/resources/attention-mechanism-innovation-next-generation-focus) +- [@article@Advanced Attention Mechanisms in Transformer LLMs](https://pub.towardsai.net/advanced-attention-mechanisms-in-transformer-llms-44cac04ec356?gi=22c6a00772a8) +- [@article@​Sequential Attention: Making AI models leaner and faster without sacrificing accuracy](https://research.google/blog/sequential-attention-making-ai-models-leaner-and-faster-without-sacrificing-accuracy/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md b/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md index 448530a76..0f1108083 100644 --- a/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md +++ b/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md @@ -4,4 +4,6 @@ The Blackwell architecture, released in late 2024 with the B200 and B300, added Visit the following resources to learn more: -- [@official@NVIDIA Blackwell Architecture Technical Brief](https://resources.nvidia.com/en-us-blackwell-architecture) \ No newline at end of file +- [@official@NVIDIA Blackwell Architecture Technical Brief](https://resources.nvidia.com/en-us-blackwell-architecture) +- [@official@NVIDIA Blackwell Architecture](https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/) +- [@article@NVIDIA Blackwell GPUs: Architecture, Features, Specs](https://www.nexgencloud.com/blog/performance-benchmarks/nvidia-blackwell-gpus-architecture-features-specs) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/bottleneck-analysis@Gtd4g4MtB9zFfahRpbE2K.md b/roadmaps/inference-engineering/content/bottleneck-analysis@Gtd4g4MtB9zFfahRpbE2K.md index 243426626..4e20511fe 100644 --- a/roadmaps/inference-engineering/content/bottleneck-analysis@Gtd4g4MtB9zFfahRpbE2K.md +++ b/roadmaps/inference-engineering/content/bottleneck-analysis@Gtd4g4MtB9zFfahRpbE2K.md @@ -1,3 +1,10 @@ # Bottleneck Analysis -Bottleneck analysis identifies whether a given inference workload is limited by compute or by memory bandwidth, which determines which optimizations will actually improve performance. The core tool is the ops:byte ratio: comparing a GPU's peak FLOPS against its peak memory bandwidth reveals the hardware's balance point, and comparing a model operation's arithmetic intensity against that balance point reveals whether it is compute-bound or memory-bound. LLM prefill and image generation are compute-bound; LLM decode is memory-bound. \ No newline at end of file +Bottleneck analysis identifies whether a given inference workload is limited by compute or by memory bandwidth, which determines which optimizations will actually improve performance. The core tool is the ops:byte ratio: comparing a GPU's peak FLOPS against its peak memory bandwidth reveals the hardware's balance point, and comparing a model operation's arithmetic intensity against that balance point reveals whether it is compute-bound or memory-bound. LLM prefill and image generation are compute-bound; LLM decode is memory-bound. + +Visit the following resources to learn more: + +- [@article@The Next AI Bottleneck Isn’t the Model: It’s the Inference System](https://towardsdatascience.com/the-next-ai-bottleneck-isnt-the-model-its-the-inference-system/) +- [@article@Performance Analysis and Bottleneck Identification in AI Workflows](https://multicorewareinc.com/performance-analysis-and-bottleneck-identification-in-ai-workflows/) +- [@article@Distributed AI Inference: Why Placement Is the New Bottleneck](https://www.akamai.com/blog/cloud/distributed-ai-inference-why-placement-new-bottleneck) +- [@video@Inference Optimization Course](https://www.youtube.com/playlist?list=PLXV1pCewfApE) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md b/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md index d5a866418..46da96272 100644 --- a/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md +++ b/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md @@ -1,3 +1,10 @@ # Compute -A GPU's compute capability is determined by its Streaming Multiprocessors (SMs), each containing CUDA Cores for general arithmetic and Tensor Cores for matrix multiply-accumulate operations. Tensor Cores are the primary resource for inference, executing the matmuls that make up the majority of forward pass computation. GPU compute is measured in peak Tensor Core FLOPS at a given precision, and roughly doubles with each new NVIDIA architecture generation. \ No newline at end of file +A GPU's compute capability is determined by its Streaming Multiprocessors (SMs), each containing CUDA Cores for general arithmetic and Tensor Cores for matrix multiply-accumulate operations. Tensor Cores are the primary resource for inference, executing the matmuls that make up the majority of forward pass computation. GPU compute is measured in peak Tensor Core FLOPS at a given precision, and roughly doubles with each new NVIDIA architecture generation. + +Visit the following resources to learn more: + +- [@article@Understanding GPU Architecture: Structure, Layers, and Performance Explained](https://www.scalecomputing.com/resources/understanding-gpu-architecture) +- [@article@What Is GPU Computing and How Does It Work? A Complete Guide](https://factory.fpt.ai/ai-insights/what-is-gpu-computing) +- [@video@Understanding NVIDIA GPU Hardware as a CUDA C Programmer](https://www.youtube.com/watch?v=1Goq8Yc3dfo) +- [@video@Fundamentals of GPU Architecture](https://www.youtube.com/playlist?list=PLxNPSjHT5qvscDTMaIAY9boOOXAJAS7y4) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md index 7daf70b49..c3bf83e07 100644 --- a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md +++ b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md @@ -6,4 +6,5 @@ Visit the following resources to learn more: - [@official@Ollama](https://ollama.com/) - [@official@LM Studio](https://lmstudio.ai/) -- [@opensource@ComfyUI](https://github.com/comfyanonymous/ComfyUI) \ No newline at end of file +- [@opensource@ComfyUI](https://github.com/comfyanonymous/ComfyUI) +- [@article@What Is Local AI Inference — and Why It Might Change How You Use AI](https://www.merciaai.com/post/what-is-local-ai-inference-and-why-it-might-change-how-you-use-ai) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md index 00589973d..04baaf483 100644 --- a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md +++ b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md @@ -4,9 +4,6 @@ Diffusion models generate images by starting from random noise and iteratively d Visit the following resources to learn more: -- [@article@Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) -- [@article@High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) -- [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) -- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) -- [@article@How DALL-E 2 Actually Works](https://www.assemblyai.com/blog/how-dall-e-2-actually-works/) +- [@article@What are diffusion models?](https://www.ibm.com/think/topics/diffusion-models) +- [@video@Diffusion Models for AI Image Generation](https://www.youtube.com/watch?v=x2GRE-RzmD8) - [@video@How AI Image Generators Work (Stable Diffusion / DALL-E)](https://www.youtube.com/watch?v=1CIpzeNxIhU) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md b/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md index a5d9240e0..745baecb4 100644 --- a/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md +++ b/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md @@ -4,7 +4,7 @@ Neural networks can be organized as encoder-only, decoder-only, or encoder-decod Visit the following resources to learn more: -- [@article@What is an encoder-decoder model?](https://www.ibm.com/think/topics/encoder-decoder-model) - [@course@Encoder-Decoder Architecture](https://www.skills.google/course_templates/543) +- [@article@What is an encoder-decoder model?](https://www.ibm.com/think/topics/encoder-decoder-model) - [@video@Encoder-decoder architecture: Overview](https://www.youtube.com/watch?v=zbdong_h-x4) - [@video@Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks,](https://www.youtube.com/watch?v=L8HKweZIOmg) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md b/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md index 86272102e..816a01069 100644 --- a/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md +++ b/roadmaps/inference-engineering/content/few-step-image-generation@biVjpCA3bsKI7sa_aBIUE.md @@ -4,4 +4,5 @@ Few-step image generation models produce usable images in eight or fewer denoisi Visit the following resources to learn more: -- [@article@Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378) \ No newline at end of file +- [@article@Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378) +- [@article@Few-Step Diffusion Models](https://www.emergentmind.com/topics/few-step-diffusion-model) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md b/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md index 4e4b8852f..6c1573e2a 100644 --- a/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md +++ b/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md @@ -6,5 +6,5 @@ Visit the following resources to learn more: - [@opensource@Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention) - [@article@FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135) -- [@article@FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://arxiv.org/abs/2307.08691) -- [@article@FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://arxiv.org/abs/2407.08608) \ No newline at end of file +- [@video@Flash Attention: The Fastest Attention Mechanism?](https://www.youtube.com/watch?v=eJn6zsbrqUM) +- [@video@FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://www.youtube.com/playlist?list=PLBWdTDczuFNrxnetcH-5CVLRP1u4Ua87m) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md b/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md index 8fe1a58f6..bb8b685a9 100644 --- a/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md +++ b/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md @@ -5,5 +5,6 @@ GPU architecture determines the hard limits of what inference can achieve on a g Visit the following resources to learn more: - [@book@Programming Massively Parallel Processors](https://www.goodreads.com/work/editions/10244675) -- [@article@NVIDIA Tesla: A Unified Graphics and Computing Architecture](https://ieeexplore.ieee.org/document/4523358) -- [@article@GPU Glossary](https://modal.com/gpu-glossary) \ No newline at end of file +- [@article@Intro to GPUs](https://max.modular.com/gpu/architecture/) +- [@article@GPU Glossary](https://modal.com/gpu-glossary) +- [@video@How do Graphics Cards Work? Exploring GPU Architecture](https://www.youtube.com/watch?v=h9Z4oGN89MU) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md b/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md index 5b843d33c..f07aaf1cb 100644 --- a/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md +++ b/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md @@ -1,3 +1,9 @@ # GPU Models -Knowing the GPU landscape — which devices are available, what they offer, and how they compare — is practical knowledge for procurement decisions, capacity planning, and understanding which optimizations apply to which hardware. This group covers the NVIDIA architectures that dominate inference deployments and the alternatives worth knowing. \ No newline at end of file +Knowing the GPU landscape — which devices are available, what they offer, and how they compare — is practical knowledge for procurement decisions, capacity planning, and understanding which optimizations apply to which hardware. This group covers the NVIDIA architectures that dominate inference deployments and the alternatives worth knowing. + +Visit the following resources to learn more: + +- [@article@12 best GPUs for AI and machine learning in 2026](https://northflank.com/blog/best-gpu-for-ai) +- [@article@GPU machine types](https://docs.cloud.google.com/compute/docs/gpus) +- [@video@Every 2026 GPU Brand Explained: What’s the Real Difference?](https://www.youtube.com/watch?v=namWLTrbA1A) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md b/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md index 48ecbf43e..afc9e99b9 100644 --- a/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md +++ b/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md @@ -4,4 +4,7 @@ NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink C Visit the following resources to learn more: -- [@official@NVIDIA Grace CPU](https://www.nvidia.com/en-us/data-center/grace-cpu/) \ No newline at end of file +- [@official@NVIDIA Grace CPU](https://www.nvidia.com/en-us/data-center/grace-cpu/) +- [@official@NVIDIA Vera CPU](https://www.nvidia.com/en-us/data-center/vera-cpu/) +- [@article@NVIDIA Vera/Rubin](https://cb4uhost.com/gpus/vera-rubin) +- [@article@NVIDIA Vera Rubin: What is it, what's new, and when you can get it](https://www.civo.com/blog/nvidia-vera-rubin) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md b/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md index 0ef41045a..5edaf3c47 100644 --- a/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md +++ b/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md @@ -4,4 +4,7 @@ The Hopper architecture, released in 2022 with the H100 and later the H200, intr Visit the following resources to learn more: -- [@official@NVIDIA H100 Tensor Core GPU Architecture](https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c) \ No newline at end of file +- [@official@NVIDIA H100 Tensor Core GPU Architecture](https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c) +- [@official@NVIDIA Hopper Architecture](https://www.nvidia.com/en-us/data-center/technologies/hopper-architecture/) +- [@article@What is a Hopper GPU?](https://www.ionos.com/digitalguide/server/know-how/hopper-gpu/) +- [@article@Nvidia Hopper Explained: Why a 2022 GPU Is Still the Smart Money in 2026](https://www.moduledge.com/blog/nvidia-hopper) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md b/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md index 6360c5bc8..3e29a70ea 100644 --- a/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md +++ b/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md @@ -1,3 +1,10 @@ # Image and Video Generation Bottlenecks -Image and video generation models are compute-bound because each denoising step applies attention over the entire latent space in parallel, similar to LLM prefill. The models are also relatively small, so memory bandwidth is not the limiting factor. Optimizations focus on FLOPS efficiency rather than memory traffic reduction. \ No newline at end of file +Image and video generation models are compute-bound because each denoising step applies attention over the entire latent space in parallel, similar to LLM prefill. The models are also relatively small, so memory bandwidth is not the limiting factor. Optimizations focus on FLOPS efficiency rather than memory traffic reduction. + +Visit the following resources to learn more: + +- [@article@How to Overcome Visual Data Bottlenecks in GenAI Development](https://wirestock.io/gen-ai-resources/how-to-overcome-visual-data-bottlenecks-in-genai-development) +- [@article@6 Limitations of Current AI Video Generation Technology and Workarounds](https://kling.ai/blog/limitations-of-current-ai-video-generation-technology) +- [@article@The biggest bottleneck in AI video generation is not the model, but the cost.](https://en.eeworld.com.cn/news/wltx/eic737186.html) +- [@video@But how do AI images and videos actually work?](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=315s) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/image--video-models@Mebx7HS-I-eNP5i3f6Clf.md b/roadmaps/inference-engineering/content/image--video-models@Mebx7HS-I-eNP5i3f6Clf.md index 39bb76c16..2ac4c149f 100644 --- a/roadmaps/inference-engineering/content/image--video-models@Mebx7HS-I-eNP5i3f6Clf.md +++ b/roadmaps/inference-engineering/content/image--video-models@Mebx7HS-I-eNP5i3f6Clf.md @@ -1,3 +1,10 @@ # Image & Video Models -Image and video models are specialized artificial intelligence architectures designed to process, interpret, and generate visual data. These models utilize techniques like convolutional neural networks (CNNs), transformers, and diffusion processes to analyze pixel patterns for tasks such as object detection, image classification, and video frame synthesis. \ No newline at end of file +Image and video models are specialized artificial intelligence architectures designed to process, interpret, and generate visual data. These models utilize techniques like convolutional neural networks (CNNs), transformers, and diffusion processes to analyze pixel patterns for tasks such as object detection, image classification, and video frame synthesis. + +Visit the following resources to learn more: + +- [@article@Understanding AI Image Generation: Models, Tools, and Techniques](https://www.digitalocean.com/community/tutorials/understanding-ai-image-generation-models-tools-and-techniques) +- [@article@How do AI models generate videos?](https://www.technologyreview.com/2025/09/12/1123562/how-do-ai-models-generate-videos/) +- [@video@What Are Vision Language Models? How AI Sees & Understands Images](https://www.youtube.com/watch?v=lOD_EE96jhM) +- [@video@But how do AI images and videos actually work?](https://www.youtube.com/watch?v=iv-5mZ_9CPY) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md b/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md index 540eff74d..c9864d92c 100644 --- a/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md +++ b/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md @@ -4,7 +4,5 @@ Image generation models are pipelines of multiple components: a text encoder tha Visit the following resources to learn more: -- [@article@Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) -- [@article@High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) -- [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) -- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) \ No newline at end of file +- [@article@Image Generation Guide](https://ai-guide.future.mozilla.org/content/image-models/) +- [@video@How AI Image Generators Work (Stable Diffusion / Dall-E)](https://www.youtube.com/watch?v=1CIpzeNxIhU) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md b/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md index ab514447e..3b917a606 100644 --- a/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md +++ b/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md @@ -5,4 +5,6 @@ Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, Visit the following resources to learn more: - [@official@NVLink and NVSwitch](https://www.nvidia.com/en-us/data-center/nvlink/) -- [@official@NVIDIA InfiniBand](https://www.nvidia.com/en-us/networking/products/infiniband/) \ No newline at end of file +- [@official@NVIDIA InfiniBand](https://www.nvidia.com/en-us/networking/products/infiniband/) +- [@article@Instances](https://www.baseten.co/inference-engineering/book/03-hardware/instances/) +- [@video@GPUs: Explained](https://www.youtube.com/watch?v=LfdK-v0SbGI) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md b/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md index 7afed5338..94fc7fd6a 100644 --- a/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md +++ b/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md @@ -4,4 +4,7 @@ The KV cache stores the key and value tensors computed during attention for each Visit the following resources to learn more: -- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) \ No newline at end of file +- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) +- [@article@KV Caching Explained: Optimizing Transformer Inference Efficiency](https://huggingface.co/blog/not-lain/kv-caching) +- [@video@How KV Cache Speeds Up LLMs for Faster AI Models on GPUs](https://www.youtube.com/watch?v=o0gkdZBtwEg) +- [@video@The KV Cache: Memory Usage in Transformers](https://www.youtube.com/watch?v=80bIUggRJf4) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md index 98dc801a9..206aaa97b 100644 --- a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md +++ b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md @@ -4,7 +4,7 @@ An LLM is a decoder-only transformer with three main components: an embedding la Visit the following resources to learn more: -- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) +- [@book@Build a Large Language Model From Scratch](https://dl.tehrandata.org/book/Build-a-large-language-models.pdf) - [@article@What is a Large Language Model (LLM)?](https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/) - [@article@New to LLMs? Start Here](https://towardsdatascience.com/new-to-llms-start-here/?utm_source=roadmap&utm_medium=Referral&utm_campaign=TDS+roadmap+integration) - [@video@How Large Language Models Work](https://www.youtube.com/watch?v=5sLYAQS9sWQ) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md b/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md index a48e32754..f9a5c3693 100644 --- a/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md +++ b/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md @@ -5,4 +5,6 @@ LLM prefill is compute-bound because it processes the entire input sequence as o Visit the following resources to learn more: - [@book@AI Systems Performance Engineering](https://www.oreilly.com/library/view/ai-systems-performance/9781098170202/) -- [@article@GPU Glossary](https://modal.com/gpu-glossary) \ No newline at end of file +- [@article@Common Bottlenecks in LLM Inference at Scale (And How to Fix Them)](https://www.yottalabs.ai/post/common-bottlenecks-in-llm-inference-at-scale-and-how-to-fix-them) +- [@article@GPU Glossary](https://modal.com/gpu-glossary) +- [@article@The Hidden Bottlenecks in LLM Inference and How to Fix Them](https://www.digitalocean.com/community/conceptual-articles/bottlenecks-llm-inference-optimization) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md b/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md index a2e2dd7b4..7e3fa68f1 100644 --- a/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md +++ b/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md @@ -1,3 +1,10 @@ # Local and Edge Inference -Not all inference runs in a datacenter. Desktop and mobile deployments have different hardware constraints, tooling ecosystems, and use cases from cloud inference. This group covers the patterns and tools for running models on consumer hardware and mobile devices. \ No newline at end of file +Not all inference runs in a datacenter. Desktop and mobile deployments have different hardware constraints, tooling ecosystems, and use cases from cloud inference. This group covers the patterns and tools for running models on consumer hardware and mobile devices. + +Visit the following resources to learn more: + +- [@article@What Is Edge Inference?](https://aws.amazon.com/what-is/edge-inference/) +- [@article@AI-Focused Edge Inference: Use Cases And Guide for Enterprise](https://www.mirantis.com/blog/ai-focused-edge-inference-use-cases-and-guide-for-enterprise/) +- [@article@What Is Local Inference? Definition & Requirements](https://ltx.io/glossary/local-inference) +- [@video@What is Edge AI?](https://www.youtube.com/watch?v=QXuvUW4B7jU&t=232s) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md b/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md index 6eb2a1b87..79637f45a 100644 --- a/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md +++ b/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md @@ -1,3 +1,10 @@ # Memory -GPU memory determines what fits on a single device and how fast data can move through the compute units. VRAM capacity sets the upper limit on model size and KV cache; VRAM bandwidth determines how quickly the decode phase can load weights per token. Modern datacenter GPUs use high-bandwidth memory (HBM) in generations HBM3, HBM3e, and HBM4, stacked directly on the GPU die to maximize bandwidth. \ No newline at end of file +GPU memory determines what fits on a single device and how fast data can move through the compute units. VRAM capacity sets the upper limit on model size and KV cache; VRAM bandwidth determines how quickly the decode phase can load weights per token. Modern datacenter GPUs use high-bandwidth memory (HBM) in generations HBM3, HBM3e, and HBM4, stacked directly on the GPU die to maximize bandwidth. + +Visit the following resources to learn more: + +- [@article@What is GPU Memory and Why it Matters for LLM Inference](https://www.bentoml.com/blog/what-is-gpu-memory-and-why-it-matters-for-llm-inference) +- [@article@What is GPU Memory (VRAM)?](https://www.ertas.ai/glossary/gpu-memory) +- [@video@NPU vs. CPU vs. GPU vs. TPU: AI Hardware Compared](https://www.youtube.com/watch?v=d3SqH0UBLEY&t=58s) +- [@video@Memory Hierarchy | GPU Programming](https://www.youtube.com/watch?v=Zrbw0zajhJM) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md b/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md index d1a4bde24..a1abc8ef2 100644 --- a/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md +++ b/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md @@ -4,4 +4,7 @@ Mixture of Experts models replace dense linear layers with a set of smaller expe Visit the following resources to learn more: -- [@article@Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538) \ No newline at end of file +- [@article@Mixture of Experts Explained](https://huggingface.co/blog/moe) +- [@article@Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538) +- [@video@What is Mixture of Experts?](https://www.youtube.com/watch?v=sYDlVVyJYn4) +- [@video@A Visual Guide to Mixture of Experts (MoE) in LLMs](https://www.youtube.com/watch?v=sOPDGQjFcuM) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md b/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md index 730bf2227..23ded4e0b 100644 --- a/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md +++ b/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md @@ -1,3 +1,9 @@ # Mobile Inference -Mobile devices run inference using vendor frameworks: Google's AI Edge SDK for Android and Apple's Foundation Models and Core ML for iOS. Hardware limitations restrict practical model sizes to one to two billion parameters on current phones. Small fine-tuned models for tasks like transcription, translation, and on-device classification are the strongest fit for mobile edge inference. \ No newline at end of file +Mobile devices run inference using vendor frameworks: Google's AI Edge SDK for Android and Apple's Foundation Models and Core ML for iOS. Hardware limitations restrict practical model sizes to one to two billion parameters on current phones. Small fine-tuned models for tasks like transcription, translation, and on-device classification are the strongest fit for mobile edge inference. + +Visit the following resources to learn more: + +- [@article@Shifting AI inference from the cloud to your phone can reduce AI costs](https://www.qualcomm.com/news/onq/2025/09/shifting-ai-inference-from-the-cloud-to-your-phone-can-reduce-ai-costs) +- [@article@LLM Inference guide for Android](https://developers.google.com/edge/mediapipe/solutions/genai/llm_inference/android) +- [@article@Cactus: High-Performance AI Inference on Any Smartphone](https://huggingface.co/blog/rshemet/cactus-on-device-inference) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md b/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md index 825026ed9..fb4d69de8 100644 --- a/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md +++ b/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md @@ -1,3 +1,8 @@ # NVIDIA -NVIDIA GPUs are the dominant hardware for inference due to their combination of Tensor Core compute, HBM bandwidth, NVLink interconnects, and the CUDA software ecosystem. The three most relevant architectures for current inference deployments are Hopper, Blackwell, and the Grace and Vera CPU platforms that pair with them. \ No newline at end of file +NVIDIA GPUs are the dominant hardware for inference due to their combination of Tensor Core compute, HBM bandwidth, NVLink interconnects, and the CUDA software ecosystem. The three most relevant architectures for current inference deployments are Hopper, Blackwell, and the Grace and Vera CPU platforms that pair with them. + +Visit the following resources to learn more: + +- [@article@Top 12 NVIDIA GPUs for AI Training & Inference in 2026](https://www.atlantic.net/gpu-server-hosting/top-nvidia-gpus-for-ai-training-and-inference/) +- [@video@Every Nvidia Graphics Card Explained as Simply as Possible](https://www.youtube.com/watch?v=UwPezhj2zoY) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/opsbyte-ratio@uc2enkdtRqA5K7kbgrYJF.md b/roadmaps/inference-engineering/content/opsbyte-ratio@uc2enkdtRqA5K7kbgrYJF.md index 03af22b7e..805cf152f 100644 --- a/roadmaps/inference-engineering/content/opsbyte-ratio@uc2enkdtRqA5K7kbgrYJF.md +++ b/roadmaps/inference-engineering/content/opsbyte-ratio@uc2enkdtRqA5K7kbgrYJF.md @@ -1,3 +1,8 @@ # Ops:Byte Ratio -The ops:byte ratio is a property of GPU hardware that expresses the balance between compute throughput and memory bandwidth. It is calculated by dividing peak FLOPS by peak memory bandwidth, and sets the threshold that separates compute-bound from memory-bound workloads on that device. Different GPU generations have very different ops:byte ratios, which is why an optimization that works well on one generation may have no effect on another. \ No newline at end of file +The ops:byte ratio is a property of GPU hardware that expresses the balance between compute throughput and memory bandwidth. It is calculated by dividing peak FLOPS by peak memory bandwidth, and sets the threshold that separates compute-bound from memory-bound workloads on that device. Different GPU generations have very different ops:byte ratios, which is why an optimization that works well on one generation may have no effect on another. + +Visit the following resources to learn more: + +- [@article@A guide to LLM inference and performance](https://www.baseten.co/blog/llm-transformer-inference-guide/) +- [@article@Understanding Performance](https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html#understand-perf) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md b/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md index 6453ece1e..013c8a56f 100644 --- a/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md +++ b/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md @@ -1,3 +1,9 @@ # Other Accelerators -Alternatives to NVIDIA GPUs include AMD MI350, Google TPUs, AWS Inferentia and Trainium, Cerebras WSE-3, and Groq LPUs. Each offers a specific advantage: AMD provides CUDA-compatible tooling via ROCm; TPUs integrate deeply with Google Cloud and JAX; Inferentia and Trainium offer cost advantages within AWS. All face the shared challenge of building an ecosystem to match the breadth of NVIDIA's CUDA libraries and inference engine support. \ No newline at end of file +Alternatives to NVIDIA GPUs include AMD MI350, Google TPUs, AWS Inferentia and Trainium, Cerebras WSE-3, and Groq LPUs. Each offers a specific advantage: AMD provides CUDA-compatible tooling via ROCm; TPUs integrate deeply with Google Cloud and JAX; Inferentia and Trainium offer cost advantages within AWS. All face the shared challenge of building an ecosystem to match the breadth of NVIDIA's CUDA libraries and inference engine support. + +Visit the following resources to learn more: + +- [@article@On-Device AI Without NVIDIA](https://www.buildmvpfast.com/blog/on-device-ai-without-nvidia-apple-silicon-intel-alternatives-2026) +- [@article@Top 12 Cloud GPU Providers for AI and Machine Learning in 2026](https://www.runpod.io/articles/guides/top-cloud-gpu-providers) +- [@video@Every 2026 GPU Brand Explained: What’s the Real Difference?](https://www.youtube.com/watch?v=namWLTrbA1A) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md b/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md index bb48f9fd1..012d4583a 100644 --- a/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md +++ b/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md @@ -4,4 +4,6 @@ PagedAttention stores the KV cache in fixed-size pages rather than a single cont Visit the following resources to learn more: -- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) \ No newline at end of file +- [@article@Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) +- [@article@PagedAttention](https://handbook.modular.com/inference-optimization/pagedattention/) +- [@video@PagedAttention: Behind vLLM's Insane Speed](https://www.youtube.com/watch?v=6uPnLkCiy5g) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/roofline-model@XF-wMQCqPZiWv1XmryPBP.md b/roadmaps/inference-engineering/content/roofline-model@XF-wMQCqPZiWv1XmryPBP.md index 0cebec277..2a6b5046c 100644 --- a/roadmaps/inference-engineering/content/roofline-model@XF-wMQCqPZiWv1XmryPBP.md +++ b/roadmaps/inference-engineering/content/roofline-model@XF-wMQCqPZiWv1XmryPBP.md @@ -1,3 +1,9 @@ # Roofline Model -The roofline model is a visual framework for reasoning about hardware performance limits. It plots arithmetic intensity on the x-axis against achievable performance on the y-axis, with two ceilings: a diagonal memory bandwidth ceiling for low-intensity workloads and a horizontal compute ceiling for high-intensity ones. A workload's position relative to these ceilings shows not just whether it is compute-bound or memory-bound, but how far it sits from the hardware's theoretical maximum — which tells you how much headroom remains before an optimization hits a hard limit. \ No newline at end of file +The roofline model is a visual framework for reasoning about hardware performance limits. It plots arithmetic intensity on the x-axis against achievable performance on the y-axis, with two ceilings: a diagonal memory bandwidth ceiling for low-intensity workloads and a horizontal compute ceiling for high-intensity ones. A workload's position relative to these ceilings shows not just whether it is compute-bound or memory-bound, but how far it sits from the hardware's theoretical maximum — which tells you how much headroom remains before an optimization hits a hard limit. + +Visit the following resources to learn more: + +- [@article@What is the roofline model?](https://modal.com/gpu-glossary/perf/roofline-model) +- [@article@Calculating inference bottlenecks](https://learn-inference.com/chapters/models/bottlenecks) +- [@video@A very short intro to the Roofline model](https://www.youtube.com/watch?v=IrkNZG8MJ64) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md b/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md index 544e58b69..0cc0c5d49 100644 --- a/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md +++ b/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md @@ -6,4 +6,6 @@ Visit the following resources to learn more: - [@book@Designing Data-Intensive Applications](https://dataintensive.net/) - [@book@Site Reliability Engineering](https://sre.google/books/) -- [@article@SemiAnalysis](https://semianalysis.com/) \ No newline at end of file +- [@article@SemiAnalysis](https://semianalysis.com/) +- [@article@GPU Requirements for LLM Inference: Memory, Throughput, and Sizing](https://www.onesourcecloud.net/cms/gpu-requirements-llm-inference-sizing.html) +- [@article@Inference Is the New Bottleneck: How to Plan GPU Capacity for Production AI](https://clear.ml/blog/inference-is-the-new-bottleneck-how-to-plan-gpu-capacity-for-production-ai) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md b/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md index f9e2bd2d7..daa9f855d 100644 --- a/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md +++ b/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md @@ -4,4 +4,7 @@ Transformer blocks are the repeated hidden layers of an LLM. Each block contains Visit the following resources to learn more: -- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762) \ No newline at end of file +- [@article@Attention Is All You Need](https://arxiv.org/abs/1706.03762) +- [@article@What Are Transformer Models and How Do They Work?](https://cohere.com/llmu/what-are-transformer-models) +- [@article@A Simplified Explanation of the Transformer Block](https://medium.com/@akhileshkapse/a-simplified-explanation-of-the-transformer-block-must-read-blog-for-nlp-enthusiasts-12ef240a62ac) +- [@video@Transformers, the tech behind LLMs](https://www.youtube.com/watch?v=wjZofJX0v4M&t=183s) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md b/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md index b562614ff..6b5e41fa2 100644 --- a/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md +++ b/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md @@ -5,4 +5,6 @@ Video generation models extend image generation to three-dimensional latent spac Visit the following resources to learn more: - [@article@Video Diffusion Models](https://arxiv.org/abs/2204.03458) -- [@article@Imagen Video: High Definition Video Generation with Diffusion Models](https://arxiv.org/abs/2210.02303) \ No newline at end of file +- [@article@How do AI models generate videos?](https://www.technologyreview.com/2025/09/12/1123562/how-do-ai-models-generate-videos/) +- [@article@Demystifying Video Generation Models](https://adityang5.medium.com/demystifying-video-generation-models-cfac8dbe1913) +- [@video@How do AI video generation models work?](https://www.youtube.com/watch?v=Y5zHHoLrvAE) \ No newline at end of file