diff --git a/roadmaps/inference-engineering/content/ai-native-applications@ZqFQmNizkU6ykTMCrmTWf.md b/roadmaps/inference-engineering/content/ai-native-applications@ZqFQmNizkU6ykTMCrmTWf.md index d1fc58ff3..8803f3b1e 100644 --- a/roadmaps/inference-engineering/content/ai-native-applications@ZqFQmNizkU6ykTMCrmTWf.md +++ b/roadmaps/inference-engineering/content/ai-native-applications@ZqFQmNizkU6ykTMCrmTWf.md @@ -1,3 +1,10 @@ # AI-Native Applications - -AI-native applications are products where generative models power the core user experience, from code editors and voice agents to customer support and media generation. Each category has different inference requirements. Agents generate many inference calls per user action; chat applications prioritize TTFT; voice applications need low end-to-end latency for natural conversation. \ No newline at end of file + +AI-native applications are products where generative models power the core user experience, from code editors and voice agents to customer support and media generation. Each category has different inference requirements. Agents generate many inference calls per user action; chat applications prioritize TTFT; voice applications need low end-to-end latency for natural conversation. + +Visit the following resources to learn more: + +- [@roadmap@Visit the Dedicated AI Product Builder Roadmap](https://roadmap.sh/ai-product-builder) +- [@article@What is AI native?](https://www.ibm.com/think/topics/ai-native) +- [@article@Guide to Building AI-Native Applications](https://talkdev.com/learning-center/guide-to-building-ai-native-applications) +- [@video@How to Build an AI-Native Services Company](https://www.youtube.com/watch?v=gSNFJbgoaHI) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/asynchronous-inference@59mMKSva9vkoMxHErNXOd.md b/roadmaps/inference-engineering/content/asynchronous-inference@59mMKSva9vkoMxHErNXOd.md index 76d621229..b0bb8e29e 100644 --- a/roadmaps/inference-engineering/content/asynchronous-inference@59mMKSva9vkoMxHErNXOd.md +++ b/roadmaps/inference-engineering/content/asynchronous-inference@59mMKSva9vkoMxHErNXOd.md @@ -1,3 +1,8 @@ # Asynchronous Inference - -Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing. \ No newline at end of file + +Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing. + +Visit the following resources to learn more: + +- [@article@Using asynchronous inference in production](https://www.baseten.co/blog/using-asynchronous-inference-in-production/) +- [@article@Asynchronous Inference: Methods & Applications](https://www.emergentmind.com/topics/asynchronous-inference) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/asynchronous-inference@LAD5cttwzdbMuWIv3XjG1.md b/roadmaps/inference-engineering/content/asynchronous-inference@LAD5cttwzdbMuWIv3XjG1.md index 76d621229..08b942222 100644 --- a/roadmaps/inference-engineering/content/asynchronous-inference@LAD5cttwzdbMuWIv3XjG1.md +++ b/roadmaps/inference-engineering/content/asynchronous-inference@LAD5cttwzdbMuWIv3XjG1.md @@ -1,3 +1,3 @@ # Asynchronous Inference - + Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md b/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md index 801149e68..1aeab91ff 100644 --- a/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md +++ b/roadmaps/inference-engineering/content/attention-mechanism@2KY3T5JVHQBR8pmAiuq0V.md @@ -1,5 +1,5 @@ # Attention Mechanism - + Attention is the mechanism transformers use to relate each token to all previous tokens via query, key, and value projections. During decode, each new token attends to every prior token, and the KV cache stores precomputed keys and values to avoid recomputation. Attention is the most expensive and most heavily optimized operation in LLM inference. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md b/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md index f3aa78434..8384f037e 100644 --- a/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md +++ b/roadmaps/inference-engineering/content/attention-variants@6oTnNEsX_6Cu6lowEz9vg.md @@ -1,5 +1,5 @@ # Attention Variants - + Newer attention algorithms reduce the quadratic time complexity of standard attention. Sliding window attention limits each token to attending to the nearest w tokens, turning O(N²) into O(Nw). Linear attention approximates the softmax with a linear-time kernel. Compressed attention periodically compresses earlier context. These variants trade off some quality for better scaling on long sequences. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md b/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md index 6df8f2e60..f2cba9146 100644 --- a/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md +++ b/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md @@ -1,5 +1,5 @@ # Automatic Speech Recognition (ASR) - + ASR models like Whisper take audio input and produce text transcripts. Whisper is an encoder-decoder model where the decoder, an autoregressive transformer, dominates inference time. TensorRT-LLM with in-flight batching for the decoder delivers the best performance. For real-time transcription, streaming via WebSockets with VAD-segmented chunks achieves round-trip targets of around 200 milliseconds per chunk on optimized deployments. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md b/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md index f387ee650..fe7613462 100644 --- a/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md +++ b/roadmaps/inference-engineering/content/autoscaling@o_gotSHKIPtV5SgGN3Vja.md @@ -1,9 +1,9 @@ # Autoscaling - + Autoscaling dynamically adjusts the number of active model replicas to match incoming traffic, maintaining latency SLAs while minimizing idle GPU spend. Kubernetes is the standard orchestration layer, scaling replicas within a cluster based on traffic signals and utilization metrics. Key configuration parameters are min replicas, max replicas, the autoscaling window, the scale-down delay, and the concurrency target per replica. Visit the following resources to learn more: -- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) - [@book@Designing Data-Intensive Applications](https://dataintensive.net/) -- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file +- [@book@Site Reliability Engineering](https://sre.google/books/) +- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/benchmarking--profiling@TxK7B_FF38ms3eKPGug2w.md b/roadmaps/inference-engineering/content/benchmarking--profiling@TxK7B_FF38ms3eKPGug2w.md index 7e1a02021..e413b7e1e 100644 --- a/roadmaps/inference-engineering/content/benchmarking--profiling@TxK7B_FF38ms3eKPGug2w.md +++ b/roadmaps/inference-engineering/content/benchmarking--profiling@TxK7B_FF38ms3eKPGug2w.md @@ -1,3 +1,3 @@ # Benchmarking and Profiling - + Benchmarking and profiling are how inference engineers turn intuition into evidence. Benchmarking measures system-level throughput and latency under load; profiling zooms in to identify which specific operations are consuming time. Both are prerequisites for making optimization decisions with confidence. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md b/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md index fcb57a33b..448530a76 100644 --- a/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md +++ b/roadmaps/inference-engineering/content/blackwell@LihLABERuKBw30G_j48Fj.md @@ -1,5 +1,5 @@ # Blackwell - + The Blackwell architecture, released in late 2024 with the B200 and B300, added FP4 Tensor Cores and new microscaling number formats including MXFP8, MXFP4, and NVFP4. B200 GPUs offer approximately 5 petaFLOPS of FP8 compute and up to 8 TB/s of memory bandwidth. They are the current gold standard for demanding inference workloads including large LLMs and video generation. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/cache-aware-routing@3DR0Qvqu2qUTi9niU_63n.md b/roadmaps/inference-engineering/content/cache-aware-routing@3DR0Qvqu2qUTi9niU_63n.md index a472a923e..ff8205f5f 100644 --- a/roadmaps/inference-engineering/content/cache-aware-routing@3DR0Qvqu2qUTi9niU_63n.md +++ b/roadmaps/inference-engineering/content/cache-aware-routing@3DR0Qvqu2qUTi9niU_63n.md @@ -1,3 +1,3 @@ # Cache-Aware Routing - + In deployments with multiple replicas, cache-aware routing directs requests to the replica most likely to have a matching prefix already in its KV cache. Without this, a user in a long conversation may hit different replicas each turn, getting cache misses that increase latency. A global KV cache stored on networked SSD allows all replicas to eventually access any cached sequence. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md b/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md index 22b08af6e..cc9fa08e1 100644 --- a/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md +++ b/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md @@ -1,5 +1,5 @@ # Caching Approaches - + KV cache reuse is the primary caching strategy in LLM inference, covering both prefix caching (reusing cached KV values across requests that share a common prefix) and KV cache offloading (moving less-used cache blocks to slower storage tiers to free GPU VRAM). A related concern is cache-aware routing, which directs requests to replicas most likely to already hold the relevant cached prefix. Visit the following resources to learn more: @@ -7,4 +7,4 @@ Visit the following resources to learn more: - [@opensource@LMCache](https://github.com/LMCache/LMCache) - [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444) - [@article@What is Prompt Caching?](https://www.ibm.com/think/topics/prompt-caching) -- [@video@What is Prompt Caching? Optimize LLM Latency with AI Transformers](https://www.youtube.com/watch?v=u57EnkQaUTY) +- [@video@What is Prompt Caching? Optimize LLM Latency with AI Transformers](https://www.youtube.com/watch?v=u57EnkQaUTY) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/caching@PzEj5yq1lDQ324RWYpDdQ.md b/roadmaps/inference-engineering/content/caching@PzEj5yq1lDQ324RWYpDdQ.md index e9ba595e9..4fef12188 100644 --- a/roadmaps/inference-engineering/content/caching@PzEj5yq1lDQ324RWYpDdQ.md +++ b/roadmaps/inference-engineering/content/caching@PzEj5yq1lDQ324RWYpDdQ.md @@ -1,3 +1,3 @@ # Caching - + Caching reuses previously computed KV values to avoid redundant prefill computation across requests that share common context. It is one of the most impactful techniques for reducing TTFT in workloads with long system prompts or multi-turn conversations. This group covers the approaches, storage tiers, routing implications, and long-context strategies that make caching effective at scale. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/capacity-management@xrr5Y1sbN2eyQyuA5LlAW.md b/roadmaps/inference-engineering/content/capacity-management@xrr5Y1sbN2eyQyuA5LlAW.md index 4c929709b..bd8da7332 100644 --- a/roadmaps/inference-engineering/content/capacity-management@xrr5Y1sbN2eyQyuA5LlAW.md +++ b/roadmaps/inference-engineering/content/capacity-management@xrr5Y1sbN2eyQyuA5LlAW.md @@ -1,3 +1,3 @@ # Capacity Management - + Multi-cloud inference pools GPU capacity from multiple cloud providers and regions into a single schedulable resource, treating distinct compute pools as fungible. This requires a global control plane for deployment and scaling decisions alongside per-cluster workload planes that serve traffic and report utilization. True multi-cloud unlocks greater total capacity, improved reliability through redundancy, lower latency via geographic proximity, and data sovereignty compliance. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md b/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md index 10687e429..2ad2ba4d2 100644 --- a/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md +++ b/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md @@ -1,5 +1,5 @@ # Client Integration - + The client layer connects application code to the inference service and contributes its own slice of end-to-end latency. This group covers the connection overhead, async request patterns, and protocol choices that determine how efficiently client and server communicate. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/client-latency-overhead@gOWGIllIIxhic1NNRfugq.md b/roadmaps/inference-engineering/content/client-latency-overhead@gOWGIllIIxhic1NNRfugq.md index 5d3d9b398..347f69842 100644 --- a/roadmaps/inference-engineering/content/client-latency-overhead@gOWGIllIIxhic1NNRfugq.md +++ b/roadmaps/inference-engineering/content/client-latency-overhead@gOWGIllIIxhic1NNRfugq.md @@ -1,3 +1,3 @@ # Client Latency Overhead - + Establishing a new connection between a client and inference server takes tens of milliseconds for TLS handshake, which can represent ten percent or more of a tight latency budget. Session reuse across requests eliminates this overhead for returning clients. Standard SDKs like the OpenAI SDK handle session reuse automatically; custom clients must implement it explicitly. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/cold-starts@Q3zM9MN5lmHRAUWjdLdXR.md b/roadmaps/inference-engineering/content/cold-starts@Q3zM9MN5lmHRAUWjdLdXR.md index f0702c92a..485796f92 100644 --- a/roadmaps/inference-engineering/content/cold-starts@Q3zM9MN5lmHRAUWjdLdXR.md +++ b/roadmaps/inference-engineering/content/cold-starts@Q3zM9MN5lmHRAUWjdLdXR.md @@ -1,3 +1,3 @@ # Cold Starts - + A cold start is the time from zero replicas to serving the first request, including GPU provisioning, container image loading, model weight loading, and inference engine startup or compilation. Reducing cold start time is essential for autoscaling systems to scale down confidently and scale up responsively. Key techniques include quantized weights, local weight caching near the GPU instances, and caching compiled inference engines to disk. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md b/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md index 59ee7e0af..d5a866418 100644 --- a/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md +++ b/roadmaps/inference-engineering/content/compute@o_tknzA4tqcpbU6t5HO-T.md @@ -1,3 +1,3 @@ # Compute - + A GPU's compute capability is determined by its Streaming Multiprocessors (SMs), each containing CUDA Cores for general arithmetic and Tensor Cores for matrix multiply-accumulate operations. Tensor Cores are the primary resource for inference, executing the matmuls that make up the majority of forward pass computation. GPU compute is measured in peak Tensor Core FLOPS at a given precision, and roughly doubles with each new NVIDIA architecture generation. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/consumer-vs-b2b@SFMvVdgTZv_POWFGWWM_s.md b/roadmaps/inference-engineering/content/consumer-vs-b2b@SFMvVdgTZv_POWFGWWM_s.md index c0992a633..b3bfb12fe 100644 --- a/roadmaps/inference-engineering/content/consumer-vs-b2b@SFMvVdgTZv_POWFGWWM_s.md +++ b/roadmaps/inference-engineering/content/consumer-vs-b2b@SFMvVdgTZv_POWFGWWM_s.md @@ -1,3 +1,7 @@ # Consumer vs. B2B Workloads - -Consumer applications are cost-sensitive and have unpredictable traffic spikes, requiring flexible autoscaling and low marginal cost. Business-to-business applications prioritize uptime and consistent low latency, as they often sit in the critical path of revenue-generating workflows. Compliance requirements, including data residency and regulatory certifications, are more common in enterprise deployments. \ No newline at end of file + +Consumer applications are cost-sensitive and have unpredictable traffic spikes, requiring flexible autoscaling and low marginal cost. Business-to-business applications prioritize uptime and consistent low latency, as they often sit in the critical path of revenue-generating workflows. Compliance requirements, including data residency and regulatory certifications, are more common in enterprise deployments. + +Visit the following resources to learn more: + +- [@article@Consumer vs. B2B](https://www.baseten.co/inference-engineering/book/01-prerequisites/about-your-app/#consumer-versus-b2b) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md b/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md index 050dc2c49..46e0f05f1 100644 --- a/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md +++ b/roadmaps/inference-engineering/content/containerization@--ohScNmF5LR04P8e95AP.md @@ -1,7 +1,10 @@ # Containerization - + Containerization packages an inference service and all its dependencies into a Docker image that runs consistently across environments. Inference containers include the CUDA toolkit, Python packages, the inference engine, and any system libraries like ffmpeg for audio processing. Using a pinned dependency tree prevents breaking changes from affecting running services and ensures reproducible builds. Visit the following resources to learn more: -- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/) \ No newline at end of file +- [@official@Docker Documentation](https://docs.docker.com/) +- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/) +- [@article@What is containerization?](https://www.ibm.com/think/topics/containerization) +- [@video@Containerization Explained](https://www.youtube.com/watch?v=0qotVMX-J5s) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/continuous-batching@h_PDOMA-V-n7Pab3epuP3.md b/roadmaps/inference-engineering/content/continuous-batching@h_PDOMA-V-n7Pab3epuP3.md index f8251ba5f..b83931541 100644 --- a/roadmaps/inference-engineering/content/continuous-batching@h_PDOMA-V-n7Pab3epuP3.md +++ b/roadmaps/inference-engineering/content/continuous-batching@h_PDOMA-V-n7Pab3epuP3.md @@ -1,3 +1,3 @@ # Continuous Batching - + Continuous batching operates at the token level, swapping requests in and out of the batch as they complete individual tokens rather than waiting for a full batch to finish. This keeps GPU utilization high and minimizes the latency penalty of batching compared to static or dynamic batching. All major inference engines implement continuous batching by default. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md b/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md index dfab33158..d01d69954 100644 --- a/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md +++ b/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md @@ -1,5 +1,5 @@ # Cost Estimation - + Dedicated inference cost depends on batch sizing, traffic patterns, and sequence lengths rather than simply on token count. Comparing dedicated versus API costs requires converting per-token API pricing into total cost over a representative time window and comparing it to GPU-hour costs at the same traffic volume. Total cost of ownership also includes engineering time for building and maintaining the inference system. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md b/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md index fb437281b..6cca44c90 100644 --- a/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md +++ b/roadmaps/inference-engineering/content/cuda@DwJJLFLPHGn9zdccTlLY2.md @@ -1,9 +1,9 @@ # CUDA - + CUDA is NVIDIA's proprietary platform for programming GPUs, consisting of a kernel language (C++), a driver for hardware management, and a runtime API for launching kernels. Every inference engine, framework, and library in the NVIDIA ecosystem depends on CUDA. Understanding CUDA is the foundation for diagnosing performance problems that tools and engines cannot solve automatically. Visit the following resources to learn more: +- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example) - [@official@CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/) -- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/) -- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example) \ No newline at end of file +- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md b/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md index cf672ef34..2e1272121 100644 --- a/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md +++ b/roadmaps/inference-engineering/content/dependency-management@wEaM8VyAo6MFsc3RXQ7Ar.md @@ -1,5 +1,5 @@ # Dependency Management - + Inference dependency chains are long and fragile due to the tight coupling between CUDA versions, framework versions, and hardware architectures. Pinning exact versions of every dependency prevents unexpected breakage when upstream packages release updates. Containers for newly released models often initially depend on pre-release software builds and need to be rebuilt on stable releases within days of the model launch. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md index 52e3ddc9d..7daf70b49 100644 --- a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md +++ b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md @@ -1,5 +1,5 @@ # Desktop Inference - + Desktop inference runs on consumer hardware, primarily Apple M-series unified memory systems and high-end NVIDIA consumer GPUs. Apple hardware offers very large unified memory capacity at moderate bandwidth, enabling larger models than discrete consumer GPUs; NVIDIA consumer GPUs offer higher bandwidth with smaller capacity. Tools like Ollama and llama.cpp make it practical to run aggressively quantized models on consumer hardware with minimal setup. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md index 2348fa264..00589973d 100644 --- a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md +++ b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md @@ -1,5 +1,5 @@ # Diffusion Models - + Diffusion models generate images by starting from random noise and iteratively denoising it over 30 to 50 steps, guided by a text prompt. Each step runs the denoiser twice, once with conditioning and once without, to apply classifier-free guidance. The entire process operates in a low-dimensional latent space rather than full pixel space to make attention computationally feasible. Visit the following resources to learn more: @@ -9,4 +9,4 @@ Visit the following resources to learn more: - [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) - [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) - [@article@How DALL-E 2 Actually Works](https://www.assemblyai.com/blog/how-dall-e-2-actually-works/) -- [@video@How AI Image Generators Work (Stable Diffusion / DALL-E)](https://www.youtube.com/watch?v=1CIpzeNxIhU) +- [@video@How AI Image Generators Work (Stable Diffusion / DALL-E)](https://www.youtube.com/watch?v=1CIpzeNxIhU) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md b/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md index 2ea1e4cdc..a408f3e70 100644 --- a/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md +++ b/roadmaps/inference-engineering/content/disaggregation@f6PWIqf4LubR7lQztaoB5.md @@ -1,5 +1,5 @@ # Disaggregation - + Disaggregation separates the prefill and decode phases onto independent GPU instances that can scale separately. The prefill engine processes the input sequence and sends the resulting KV cache to the decode engine via the hardware interconnect. This removes the resource contention between compute-heavy prefill and memory-heavy decode. Disaggregation is most effective at high traffic volumes, for large models, and with prefill-heavy workloads like code editors. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md b/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md index 76082df12..17f40eb2a 100644 --- a/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md +++ b/roadmaps/inference-engineering/content/distillation@IdNkeL9pzy4lFv17ak1Vy.md @@ -1,7 +1,11 @@ # Distillation - + Distillation trains a smaller student model to emulate a larger teacher model by exposing it to the teacher's probability distributions, not just its final outputs. Unlike fine-tuning on synthetic data, distillation transfers the teacher's reasoning behavior. The resulting student model is smaller and faster while retaining much of the teacher's quality on the same tasks. Visit the following resources to learn more: -- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) \ No newline at end of file +- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) +- [@article@How Knowledge Distillation Works and When to Use It](https://www.arcee.ai/blog/how-knowledge-distillation-works-and-when-to-use-it) +- [@article@Introduction to model distillation: Efficient knowledge transfer for AI applications](https://nebius.com/blog/posts/model-distillation-intro) +- [@video@Model Distillation: How to Steal a Billion-Dollar AI](https://www.youtube.com/watch?v=QBG6bXiOq40) +- [@video@Knowledge Distillation - Explained](https://www.youtube.com/watch?v=Pwo1bLDB-jA) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md b/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md index f5c20953f..abcf98057 100644 --- a/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md +++ b/roadmaps/inference-engineering/content/draft-target-decoding@m0EIHovHWGkWULN0a3C3v.md @@ -1,5 +1,5 @@ # Draft-Target Speculative Decoding - + Draft-target speculation uses a small draft model, typically at least ten times smaller than the target, to generate draft tokens for the target model to verify. The draft model should share the same tokenizer and general behavior as the target, often being a smaller member of the same model family. It is the simplest speculation method to set up but introduces the most overhead due to running a separate model. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md b/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md index f64f3334a..bc5a47df3 100644 --- a/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md +++ b/roadmaps/inference-engineering/content/eagle@0FZTg3_ZnTpMOQ078OOxE.md @@ -1,5 +1,5 @@ # EAGLE - + EAGLE is a purpose-built speculation method where a small draft model is trained to accept hidden states from the target model as input. By seeing the target's internal representations, EAGLE achieves higher acceptance rates and longer draft sequences (up to eight tokens) than using a general-purpose small model. It integrates into the same forward pass as the target model, eliminating the orchestration overhead of separate draft and target model runs. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/embedding-model-inference@GmCkfSeVDJ8T8WLZ7DoOp.md b/roadmaps/inference-engineering/content/embedding-model-inference@GmCkfSeVDJ8T8WLZ7DoOp.md index d3e4b1c27..5f96e9384 100644 --- a/roadmaps/inference-engineering/content/embedding-model-inference@GmCkfSeVDJ8T8WLZ7DoOp.md +++ b/roadmaps/inference-engineering/content/embedding-model-inference@GmCkfSeVDJ8T8WLZ7DoOp.md @@ -1,3 +1,3 @@ # Embedding Model Inference - + TensorRT-LLM with its optimized XQA attention kernel offers the best performance for embedding models with LLM backbones. FP8 quantization of weights provides performance gains with minimal quality loss, verified by checking cosine similarity between quantized and original output vectors. Embedding models are small enough that horizontal scaling across individual GPU replicas is more effective than multi-GPU parallelism. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md b/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md index 4b4c01edf..0a8363dc3 100644 --- a/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md +++ b/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md @@ -1,5 +1,5 @@ # Embedding Models - + Embedding models convert variable-length text or image input into a fixed-length vector that encodes semantic meaning. These vectors are used in RAG, search, recommendation, and agent memory systems. Embedding inference has two distinct traffic profiles: high-throughput backfills for bulk indexing and low-latency lookups for real-time user queries, which are best served by separate deployments. Visit the following resources to learn more: @@ -9,4 +9,4 @@ Visit the following resources to learn more: - [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084) - [@article@Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147) - [@article@MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316) -- [@article@What Are Embedding Models? Benefits and Best Practices](https://cohere.com/blog/embedding-models) +- [@article@What Are Embedding Models? Benefits and Best Practices](https://cohere.com/blog/embedding-models) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md b/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md index b08efba0c..a5d9240e0 100644 --- a/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md +++ b/roadmaps/inference-engineering/content/encoder-decoder-arch@nORbxBOAtl9sNU4sIJgP3.md @@ -1,3 +1,10 @@ # Encoder-Decoder Architectures - -Neural networks can be organized as encoder-only, decoder-only, or encoder-decoder models. Modern LLMs are decoder-only. Encoder-only models like BERT are used for text embeddings and classification. Encoder-decoder models like Whisper use an encoder to process input into an internal representation and a decoder to generate output tokens autoregressively. Each architecture has different inference characteristics and tooling requirements. \ No newline at end of file + +Neural networks can be organized as encoder-only, decoder-only, or encoder-decoder models. Modern LLMs are decoder-only. Encoder-only models like BERT are used for text embeddings and classification. Encoder-decoder models like Whisper use an encoder to process input into an internal representation and a decoder to generate output tokens autoregressively. Each architecture has different inference characteristics and tooling requirements. + +Visit the following resources to learn more: + +- [@article@What is an encoder-decoder model?](https://www.ibm.com/think/topics/encoder-decoder-model) +- [@course@Encoder-Decoder Architecture](https://www.skills.google/course_templates/543) +- [@video@Encoder-decoder architecture: Overview](https://www.youtube.com/watch?v=zbdong_h-x4) +- [@video@Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks,](https://www.youtube.com/watch?v=L8HKweZIOmg) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/end-to-end-metrics@tENGACXKSKVPmZy9a3D2z.md b/roadmaps/inference-engineering/content/end-to-end-metrics@tENGACXKSKVPmZy9a3D2z.md index 24403aa2d..637523aff 100644 --- a/roadmaps/inference-engineering/content/end-to-end-metrics@tENGACXKSKVPmZy9a3D2z.md +++ b/roadmaps/inference-engineering/content/end-to-end-metrics@tENGACXKSKVPmZy9a3D2z.md @@ -1,3 +1,7 @@ # End-to-End Metrics - -End-to-end latency includes inference time, network round-trip, queue time, and client-side overhead. Inference-only metrics tell you how well the model server performs; end-to-end metrics tell you what users actually experience. When inference time is fast but end-to-end time is slow, the bottleneck is in infrastructure or client code, not the GPU. \ No newline at end of file + +End-to-end latency includes inference time, network round-trip, queue time, and client-side overhead. Inference-only metrics tell you how well the model server performs; end-to-end metrics tell you what users actually experience. When inference time is fast but end-to-end time is slow, the bottleneck is in infrastructure or client code, not the GPU. + +Visit the following resources to learn more: + +- [@article@10 LLM Inference Metrics Every AI Engineer Must Know](https://www.intoai.pub/p/llm-inference-metrics) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md b/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md index c4844f5f8..db6094f2f 100644 --- a/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md +++ b/roadmaps/inference-engineering/content/expert-parallelism@62y78y-RXZT_QCUfd-YFE.md @@ -1,5 +1,5 @@ # Expert Parallelism - + Expert Parallelism distributes MoE experts across GPUs so each GPU hosts multiple complete experts. Tokens are routed to the GPU holding their activated experts, and each expert runs independently within a single GPU without splitting across multiple. EP has lower inter-GPU communication overhead than Tensor Parallelism and scales well to multi-node deployments via InfiniBand. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md b/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md index ff4bd5aca..3e08ce12e 100644 --- a/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md +++ b/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md @@ -1,9 +1,9 @@ # Fine-Tuning - + Fine-tuning takes a pretrained foundation model and adapts it to a specific domain by training on additional labeled data. A small fine-tuned model can match a much larger general model on constrained tasks like text-to-SQL. Smaller models are faster and cheaper to serve, so fine-tuning is one of the highest-leverage tools for hitting latency and cost targets. Visit the following resources to learn more: -- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) - [@article@What is Fine-Tuning?](https://www.ibm.com/think/topics/fine-tuning) - [@article@How I Fine-Tuned Granite-Vision 2B to Beat a 90B Model](https://towardsdatascience.com/how-i-fine-tuned-granite-vision-2b-to-beat-a-90b-model-insights-and-lessons-learned/?utm_source=roadmap&utm_medium=Referral&utm_campaign=TDS+roadmap+integration) +- [@video@Is Fine-Tuning Still Needed? LLMs, RAG, & LoRA](https://www.youtube.com/watch?v=-W2JdSl1v48) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md b/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md index 655ffbbed..4e4b8852f 100644 --- a/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md +++ b/roadmaps/inference-engineering/content/flashattention@mQcj-H3R789o6nDh8tG9k.md @@ -1,5 +1,5 @@ # FlashAttention - + FlashAttention is a series of hand-optimized CUDA kernels that compute attention with fewer reads and writes to GPU memory by fusing the attention algorithm into a single kernel. Standard attention stores intermediate matrices to memory between steps; FlashAttention eliminates those round-trips. FlashAttention 3 targets Hopper GPUs and FlashAttention 4 targets Blackwell GPUs. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/frameworks--libraries@D4uNOZgG0w-nFL3a89pDM.md b/roadmaps/inference-engineering/content/frameworks--libraries@D4uNOZgG0w-nFL3a89pDM.md index 185ba62e1..444bf9680 100644 --- a/roadmaps/inference-engineering/content/frameworks--libraries@D4uNOZgG0w-nFL3a89pDM.md +++ b/roadmaps/inference-engineering/content/frameworks--libraries@D4uNOZgG0w-nFL3a89pDM.md @@ -1,3 +1,3 @@ # Frameworks and Libraries - + Frameworks and libraries sit above CUDA and below inference engines, providing the abstractions that inference engineers interact with most directly. This group covers PyTorch and the Hugging Face ecosystem that supply the model implementations and tooling the rest of the stack depends on. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/geo-aware-load-balancing@mzF2mHSj8cvr9mvr16R0N.md b/roadmaps/inference-engineering/content/geo-aware-load-balancing@mzF2mHSj8cvr9mvr16R0N.md index 0f443b46d..f189c190a 100644 --- a/roadmaps/inference-engineering/content/geo-aware-load-balancing@mzF2mHSj8cvr9mvr16R0N.md +++ b/roadmaps/inference-engineering/content/geo-aware-load-balancing@mzF2mHSj8cvr9mvr16R0N.md @@ -1,3 +1,3 @@ # Geo-Aware Load Balancing - + A global load balancer routes requests to the GPU cluster geographically closest to the requesting user. Network latency accumulates at roughly 5 milliseconds per time zone traversed, so routing a request from Singapore to San Francisco adds 50 to 100 milliseconds of unnecessary overhead. Geo-aware routing ensures that spare capacity in a distant region is used only when nearby capacity is exhausted. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md b/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md index 1b22c166f..8fe1a58f6 100644 --- a/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md +++ b/roadmaps/inference-engineering/content/gpu-architecture@00JLMvn8DUJBCw6-BxO0y.md @@ -1,9 +1,9 @@ # GPU Architecture - + GPU architecture determines the hard limits of what inference can achieve on a given device. This group covers compute, memory, interconnects, and sizing — the four dimensions an inference engineer needs to understand to make informed decisions about hardware selection, parallelism strategy, and optimization priorities. Visit the following resources to learn more: +- [@book@Programming Massively Parallel Processors](https://www.goodreads.com/work/editions/10244675) - [@article@NVIDIA Tesla: A Unified Graphics and Computing Architecture](https://ieeexplore.ieee.org/document/4523358) -- [@article@GPU Glossary](https://modal.com/gpu-glossary) -- [@book@Programming Massively Parallel Processors](https://www.goodreads.com/work/editions/10244675) \ No newline at end of file +- [@article@GPU Glossary](https://modal.com/gpu-glossary) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md b/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md index 2ca947b82..5b843d33c 100644 --- a/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md +++ b/roadmaps/inference-engineering/content/gpu-models@umz3jScWQViO9-lACUU4f.md @@ -1,3 +1,3 @@ # GPU Models - + Knowing the GPU landscape — which devices are available, what they offer, and how they compare — is practical knowledge for procurement decisions, capacity planning, and understanding which optimizations apply to which hardware. This group covers the NVIDIA architectures that dominate inference deployments and the alternatives worth knowing. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/gpu-procurement@4mbGBgTu8AtrFHVYiffkB.md b/roadmaps/inference-engineering/content/gpu-procurement@4mbGBgTu8AtrFHVYiffkB.md index 64705e8bc..ab456369c 100644 --- a/roadmaps/inference-engineering/content/gpu-procurement@4mbGBgTu8AtrFHVYiffkB.md +++ b/roadmaps/inference-engineering/content/gpu-procurement@4mbGBgTu8AtrFHVYiffkB.md @@ -1,3 +1,3 @@ # GPU Procurement - + Cloud GPUs are available from hyperscalers like AWS and GCP, GPU-focused neoclouds like CoreWeave and Nebius, and secondary markets. Procurement mechanisms include reserved instances for predictable baseline load at discounted rates, on-demand instances for flexibility, and spot instances for cost-sensitive batch workloads that tolerate preemption. Large-scale deployments typically combine all three across multiple providers. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md b/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md index 0112073f7..48ecbf43e 100644 --- a/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md +++ b/roadmaps/inference-engineering/content/grace--vera@kPT86kA0WwE_fWUAAKCTT.md @@ -1,5 +1,5 @@ # Grace and Vera CPUs - + NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink Chip-to-Chip at up to 900 GB/s, far higher than standard PCIe connections. This high-bandwidth CPU-GPU link makes offloading KV caches and LoRA weights to CPU memory practical, since data can be retrieved quickly enough to remain on the critical path. Grace pairs with Hopper and Blackwell; Vera pairs with the upcoming Rubin architecture. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md b/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md index 88ff4a4f2..0ef41045a 100644 --- a/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md +++ b/roadmaps/inference-engineering/content/hopper@XQMbaW1tFdO8Os6mxcR2M.md @@ -1,5 +1,5 @@ # Hopper - + The Hopper architecture, released in 2022 with the H100 and later the H200, introduced FP8 Tensor Cores offering twice the compute throughput of FP16 and asynchronous data transfer features used by FlashAttention 3. H100 and H200 GPUs remain among the most widely deployed inference accelerators due to their maturity, broad software support, and relative availability compared to newer generations. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md b/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md index 68ef64a65..6360c5bc8 100644 --- a/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md +++ b/roadmaps/inference-engineering/content/image--video-bottlenecks@0lRP9cmoS-_wxQSsIcRQg.md @@ -1,3 +1,3 @@ # Image and Video Generation Bottlenecks - + Image and video generation models are compute-bound because each denoising step applies attention over the entire latent space in parallel, similar to LLM prefill. The models are also relatively small, so memory bandwidth is not the limiting factor. Optimizations focus on FLOPS efficiency rather than memory traffic reduction. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/image--video-generation@U0l_xCUiizbJfjfrlBaJ0.md b/roadmaps/inference-engineering/content/image--video-generation@U0l_xCUiizbJfjfrlBaJ0.md index ebd8c2f21..de1eaeae1 100644 --- a/roadmaps/inference-engineering/content/image--video-generation@U0l_xCUiizbJfjfrlBaJ0.md +++ b/roadmaps/inference-engineering/content/image--video-generation@U0l_xCUiizbJfjfrlBaJ0.md @@ -1,3 +1,3 @@ # Image and Video Generation - + Image and video generation inference is compute-bound rather than memory-bound, making it a different optimization problem from LLM serving. This group covers the production patterns, kernel choices, and parallelism techniques specific to diffusion model deployment. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md b/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md index 3f2b22ff0..540eff74d 100644 --- a/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md +++ b/roadmaps/inference-engineering/content/image-generation-arch@xsV4WevPzhfydyvxVzGcw.md @@ -1,5 +1,5 @@ # Image Generation Architecture - + Image generation models are pipelines of multiple components: a text encoder that converts the prompt into conditioning, a denoising model that iteratively refines a latent representation from noise into an image, and a variational autoencoder that converts the final latent to pixels. The denoising model is the largest and most compute-intensive component and runs up to 100 forward passes per image generation. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md b/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md index 18a6ee6c0..b4453a406 100644 --- a/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md +++ b/roadmaps/inference-engineering/content/image-generation-inference@lnIHjq4nvCs3zvUln-jHR.md @@ -1,5 +1,5 @@ # Image Generation Inference - + Production image generation inference uses SGLang Diffusion, TensorRT, or hand-optimized PyTorch. The most important optimization target is the attention kernel, where FlashAttention 3 or 4 outperforms the default on Hopper and Blackwell respectively. GEMM kernels for linear layers can be quantized to FP8 safely. Torch compilation with manual kernel plugins caches the compiled engine to avoid multi-minute recompilation on every cold start. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/indep-component-scaling@SKs0yuBEAyeTPQxLll3EM.md b/roadmaps/inference-engineering/content/indep-component-scaling@SKs0yuBEAyeTPQxLll3EM.md index aa63f1baf..872ee8dfa 100644 --- a/roadmaps/inference-engineering/content/indep-component-scaling@SKs0yuBEAyeTPQxLll3EM.md +++ b/roadmaps/inference-engineering/content/indep-component-scaling@SKs0yuBEAyeTPQxLll3EM.md @@ -1,3 +1,3 @@ # Independent Component Scaling - + Multi-model pipelines, like a voice agent that runs VAD, ASR, an LLM, and TTS in sequence, have components with different hardware requirements and traffic patterns. Each component should autoscale independently based on its own load. Running all components within the same Kubernetes cluster minimizes inter-component network latency, which can otherwise consume a significant fraction of a tight latency budget. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/inference-engines@01j1KkEB-nIXxP-rENcJ3.md b/roadmaps/inference-engineering/content/inference-engines@01j1KkEB-nIXxP-rENcJ3.md index d336f7c04..e8c5c4c26 100644 --- a/roadmaps/inference-engineering/content/inference-engines@01j1KkEB-nIXxP-rENcJ3.md +++ b/roadmaps/inference-engineering/content/inference-engines@01j1KkEB-nIXxP-rENcJ3.md @@ -1,3 +1,3 @@ # Inference Engines - + Inference engines are the core serving layer that takes model weights and handles requests at production scale. They implement continuous batching, memory management, parallelism, and optimization techniques so engineers do not have to implement these from scratch. This group covers the three dominant LLM engines and the orchestration platform that sits above them. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md b/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md index caef5fbad..9459eab7b 100644 --- a/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md +++ b/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md @@ -1,5 +1,5 @@ # Inference vs. Training - + Training is the process of learning model weights from data. Inference is serving those weights in production to generate outputs. Training runs are compute-intensive batch jobs; inference is a continuous, latency-sensitive service. Where training success is measured in loss curves, inference success is measured in time to first token, tokens per second, and uptime. Visit the following resources to learn more: @@ -7,3 +7,4 @@ Visit the following resources to learn more: - [@article@Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165) - [@article@Inference vs Training](https://www.cloudflare.com/learning/ai/inference-vs-training/) - [@article@What is Machine Learning Inference?](https://hazelcast.com/glossary/machine-learning-inference/) +- [@video@Why Inference is hard..](https://www.youtube.com/watch?v=B18zBnjZKmc) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md b/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md index 4d70d52fe..ab514447e 100644 --- a/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md +++ b/roadmaps/inference-engineering/content/instances--interconnects@dAo4PMB_VnRFMSjyDrLj0.md @@ -1,5 +1,5 @@ # Instances and Interconnects - + Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, and NVSwitch creates an all-to-all fabric across all eight GPUs. Between nodes, InfiniBand provides inter-node connectivity at substantially lower bandwidth than NVLink. Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated slices, each with dedicated compute and memory, allowing small models to run on fractions of a high-end GPU without resource contention. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/inter-token-latency@sre_s0dvpOP0DTM66a3hu.md b/roadmaps/inference-engineering/content/inter-token-latency@sre_s0dvpOP0DTM66a3hu.md index bfdb690be..13eb6c384 100644 --- a/roadmaps/inference-engineering/content/inter-token-latency@sre_s0dvpOP0DTM66a3hu.md +++ b/roadmaps/inference-engineering/content/inter-token-latency@sre_s0dvpOP0DTM66a3hu.md @@ -1,3 +1,8 @@ # Inter-Token Latency (ITL) - -ITL is the time between successive generated tokens during decode. An ITL of 10 milliseconds corresponds to 100 tokens per second. It is the per-step measure underlying TPS and is directly tied to how quickly the GPU can complete a single decode forward pass. \ No newline at end of file + +ITL is the time between successive generated tokens during decode. An ITL of 10 milliseconds corresponds to 100 tokens per second. It is the per-step measure underlying TPS and is directly tied to how quickly the GPU can complete a single decode forward pass. + +Visit the following resources to learn more: + +- [@article@Inter-Token Latency - NVIDIA](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html#inter-token-latency) +- [@article@Inter-Token Latency](https://nhimg.org/glossary/inter-token-latency/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/introduction@GL-9aJKTJT9s4maeT37Uf.md b/roadmaps/inference-engineering/content/introduction@GL-9aJKTJT9s4maeT37Uf.md index 5b864a4b6..b3a157ca5 100644 --- a/roadmaps/inference-engineering/content/introduction@GL-9aJKTJT9s4maeT37Uf.md +++ b/roadmaps/inference-engineering/content/introduction@GL-9aJKTJT9s4maeT37Uf.md @@ -1,3 +1,10 @@ # Inference Engineer - -Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable to serve in production. It spans the full stack from CUDA kernels and GPU architecture to Kubernetes autoscaling and multi-cloud capacity management. The field emerged alongside the explosion of open models and the shift from pay-per-token APIs to dedicated deployments, where engineers own latency, throughput, cost, and uptime directly. \ No newline at end of file + +Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable to serve in production. It spans the full stack from CUDA kernels and GPU architecture to Kubernetes autoscaling and multi-cloud capacity management. The field emerged alongside the explosion of open models and the shift from pay-per-token APIs to dedicated deployments, where engineers own latency, throughput, cost, and uptime directly. + +Visit the following resources to learn more: + +- [@book@Inference Engineering](https://www.baseten.co/inference-engineering/digital-download/) +- [@article@What Is Inference Engineering? The 2026 GPU Cloud Guide](https://www.spheron.network/blog/inference-engineering-guide-2026/) +- [@article@A Guide to AI Inference Engineering](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering) +- [@video@AI Inference: The Secret to AI's Superpowers](https://www.youtube.com/watch?v=XtT5i0ZeHHE&t=19s) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/kernel-selection--fusion@iKW1wqzd5EfRKvNIRshNN.md b/roadmaps/inference-engineering/content/kernel-selection--fusion@iKW1wqzd5EfRKvNIRshNN.md index 238d77628..36f363b10 100644 --- a/roadmaps/inference-engineering/content/kernel-selection--fusion@iKW1wqzd5EfRKvNIRshNN.md +++ b/roadmaps/inference-engineering/content/kernel-selection--fusion@iKW1wqzd5EfRKvNIRshNN.md @@ -1,3 +1,3 @@ # Kernel Selection and Fusion - + A CUDA kernel is a function that runs in parallel across thousands of GPU threads simultaneously. Kernel selection is choosing the best existing kernel for a specific operation; kernel fusion combines sequential kernels into one to eliminate unnecessary reads and writes to VRAM between operations. Libraries like cuBLAS, CUTLASS, CuTe, and FlashInfer provide pre-built kernels for common inference operations, and most inference engines perform kernel selection automatically. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md b/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md index 68c41d093..55894fcab 100644 --- a/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md +++ b/roadmaps/inference-engineering/content/kv-cache-storage@gkt2Gr34RJJf1NZgbmtXf.md @@ -1,5 +1,5 @@ # KV Cache Storage - + KV caches can be stored in GPU VRAM (fastest, tens to hundreds of gigabytes), CPU RAM (slower, hundreds of gigabytes to terabytes), local SSD, or networked SSD. Moving less frequently accessed KV blocks to slower storage frees VRAM for active requests. NVIDIA Dynamo's KV Block Manager provides APIs for managing KV blocks across these storage tiers transparently. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md b/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md index b2c02c6e6..7afed5338 100644 --- a/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md +++ b/roadmaps/inference-engineering/content/kv-cache@V678kX8FPChTZCe7Nqwhv.md @@ -1,5 +1,5 @@ # KV Cache - + The KV cache stores the key and value tensors computed during attention for each previously processed token. Without it, attention would require recomputing every prior token on each decode step, making inference prohibitively slow. The KV cache lives in GPU memory and its size grows with sequence length and batch size, making it a major target for memory management techniques. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md index 70196dbe5..98dc801a9 100644 --- a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md +++ b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md @@ -4,7 +4,7 @@ An LLM is a decoder-only transformer with three main components: an embedding la Visit the following resources to learn more: +- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) - [@article@What is a Large Language Model (LLM)?](https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/) - [@article@New to LLMs? Start Here](https://towardsdatascience.com/new-to-llms-start-here/?utm_source=roadmap&utm_medium=Referral&utm_campaign=TDS+roadmap+integration) -- [@video@How Large Language Models Work](https://www.youtube.com/watch?v=5sLYAQS9sWQ) -- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) \ No newline at end of file +- [@video@How Large Language Models Work](https://www.youtube.com/watch?v=5sLYAQS9sWQ) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/latency-percentiles@8y280d5BPXrDZwtsHZCaX.md b/roadmaps/inference-engineering/content/latency-percentiles@8y280d5BPXrDZwtsHZCaX.md index 28895cbf9..db24b12c9 100644 --- a/roadmaps/inference-engineering/content/latency-percentiles@8y280d5BPXrDZwtsHZCaX.md +++ b/roadmaps/inference-engineering/content/latency-percentiles@8y280d5BPXrDZwtsHZCaX.md @@ -1,3 +1,9 @@ # Latency Percentiles - -Mean latency hides outliers. LLM response times follow a right-skewed distribution, so engineers measure P50, P90, P95, and P99. A P99 of several seconds means one in a hundred users has a poor experience even if the median is fast. Reducing P90/P99 latency is as important as reducing average latency. \ No newline at end of file + +Mean latency hides outliers. LLM response times follow a right-skewed distribution, so engineers measure P50, P90, P95, and P99. A P99 of several seconds means one in a hundred users has a poor experience even if the median is fast. Reducing P90/P99 latency is as important as reducing average latency. + +Visit the following resources to learn more: + +- [@article@Percentiles vs averages: why your latency dashboard lies](https://clickhouse.com/resources/engineering/percentiles-vs-averages) +- [@article@Latency percentiles for load testing analysis](https://gatling.io/blog/latency-percentiles-for-load-testing-analysis) +- [@video@Mastering Latency Metrics: P90, P95, P99 | System Design](https://www.youtube.com/watch?v=lJ4NEMNBeS4) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md b/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md index d769b26c5..d96f506cd 100644 --- a/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md +++ b/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md @@ -1,7 +1,10 @@ # Latency vs. Throughput - + Lower latency means faster responses per user; higher throughput means more total requests served per unit time. These goals are often in tension. Increasing batch size improves throughput but worsens per-user latency. Choosing the right balance is one of the primary decisions in configuring an inference service. Visit the following resources to learn more: -- [@article@The LLM Inference Trilemma: Throughput, Latency, Cost](https://www.digitalocean.com/blog/llm-inference-tradeoffs) \ No newline at end of file +- [@article@The LLM Inference Trilemma: Throughput, Latency, Cost](https://www.digitalocean.com/blog/llm-inference-tradeoffs) +- [@article@LLM inference latency: TTFT, tokens per second, and what to measure](https://clickhouse.com/resources/engineering/llm-inference-latency) +- [@article@⚖️ Latency vs Throughput: Deep Dive into System Performance Trade‑Offs](https://medium.com/@shivanimutke2501/%EF%B8%8F-latency-vs-throughput-deep-dive-into-system-performance-trade-offs-adcd5860dc56) +- [@video@Latency vs Throughput in 60 Seconds](https://www.youtube.com/watch?v=azUdQ9bme9g) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md b/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md index 8975a26a8..a48e32754 100644 --- a/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md +++ b/roadmaps/inference-engineering/content/llm-bottlenecks@QVadKzBfLdegat-4wgOMX.md @@ -1,8 +1,8 @@ # LLM Inference Bottlenecks - + LLM prefill is compute-bound because it processes the entire input sequence as one large matrix multiplication, performing many operations per byte loaded. LLM decode is memory-bound because it generates tokens one at a time via vector-matrix multiplications that require loading all model weights from memory for each token while performing relatively few arithmetic operations. Visit the following resources to learn more: -- [@article@GPU Glossary](https://modal.com/gpu-glossary) -- [@book@AI Systems Performance Engineering](https://www.oreilly.com/library/view/ai-systems-performance/9781098170202/) \ No newline at end of file +- [@book@AI Systems Performance Engineering](https://www.oreilly.com/library/view/ai-systems-performance/9781098170202/) +- [@article@GPU Glossary](https://modal.com/gpu-glossary) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md b/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md index c56b0e51a..a2e2dd7b4 100644 --- a/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md +++ b/roadmaps/inference-engineering/content/local--edge-inference@FilQGz3geyyvHZGyKPW3G.md @@ -1,3 +1,3 @@ # Local and Edge Inference - + Not all inference runs in a datacenter. Desktop and mobile deployments have different hardware constraints, tooling ecosystems, and use cases from cloud inference. This group covers the patterns and tools for running models on consumer hardware and mobile devices. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md b/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md index 081d6bb9d..e7d544622 100644 --- a/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md +++ b/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md @@ -1,5 +1,5 @@ # Long Context Handling - + Long sequences produce KV caches large enough to strain GPU memory, slow down decode, and increase prefill time. Techniques for handling long context include FlashAttention for memory-efficient attention computation, PagedAttention for non-contiguous KV storage, chunked prefill to interleave long prefill with decode, and model-level approaches like sliding window attention that limit how many prior tokens each token attends to. Visit the following resources to learn more: @@ -8,4 +8,4 @@ Visit the following resources to learn more: - [@article@RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864) - [@article@What is Long Context and Why Does it Matter for AI?](https://cloud.google.com/transform/the-prompt-what-are-long-context-windows-and-why-do-they-matter) - [@article@Long-Context LLM Infrastructure: Building Systems for Million-Token Windows](https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide) -- [@video@Deep Dive into Long Context](https://www.youtube.com/watch?v=NHMJ9mqKeMQ) +- [@video@Deep Dive into Long Context](https://www.youtube.com/watch?v=NHMJ9mqKeMQ) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/lora-aware-routing@KdceJEDBdy3TbzG5suE38.md b/roadmaps/inference-engineering/content/lora-aware-routing@KdceJEDBdy3TbzG5suE38.md index aea6224bb..82af7795b 100644 --- a/roadmaps/inference-engineering/content/lora-aware-routing@KdceJEDBdy3TbzG5suE38.md +++ b/roadmaps/inference-engineering/content/lora-aware-routing@KdceJEDBdy3TbzG5suE38.md @@ -1,3 +1,3 @@ # LoRA-Aware Routing - + LoRA fine-tune weights are small adapter layers loaded on top of a base model to specialize its behavior for a specific task or customer. A deployment serving many LoRA variants for the same base model must route each request to a replica that already has the required LoRA loaded in GPU memory. Routing without this awareness forces constant LoRA swapping, adding latency and reducing throughput on each affected replica. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/measuring-quality-impact@mK_4R8liXtl64iG7ONvgz.md b/roadmaps/inference-engineering/content/measuring-quality-impact@mK_4R8liXtl64iG7ONvgz.md index e16ed66a6..9e0432738 100644 --- a/roadmaps/inference-engineering/content/measuring-quality-impact@mK_4R8liXtl64iG7ONvgz.md +++ b/roadmaps/inference-engineering/content/measuring-quality-impact@mK_4R8liXtl64iG7ONvgz.md @@ -1,3 +1,3 @@ # Measuring Quality Impact - + After quantizing a model, quality must be validated against the original. Three methods are used: perplexity scoring (lower is better, with minimal increase after quantization), standard intelligence benchmarks like MMLU or SWE-bench, and product-specific evals tailored to the actual use case. All three should show differences indistinguishable from noise before a quantized model is deployed. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md b/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md index e3b0e8d59..6eb2a1b87 100644 --- a/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md +++ b/roadmaps/inference-engineering/content/memory@5829d5dvXq9qlMSD1CG0N.md @@ -1,3 +1,3 @@ # Memory - + GPU memory determines what fits on a single device and how fast data can move through the compute units. VRAM capacity sets the upper limit on model size and KV cache; VRAM bandwidth determines how quickly the decode phase can load weights per token. Modern datacenter GPUs use high-bandwidth memory (HBM) in generations HBM3, HBM3e, and HBM4, stacked directly on the GPU die to maximize bandwidth. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md b/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md index 3464eef3d..d1a4bde24 100644 --- a/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md +++ b/roadmaps/inference-engineering/content/mixture-of-experts-moe@7BddTfOkWmlL4HAkFBXtq.md @@ -1,5 +1,5 @@ # Mixture of Experts (MoE) - + Mixture of Experts models replace dense linear layers with a set of smaller expert matrices, activating only a subset of experts per token via a learned router. A model like Qwen3-235B activates only 22 billion of its 235 billion parameters per forward pass, making single-user inference efficient. In batched production serving, different tokens activate different experts, so nearly all parameters are active across the batch. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md b/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md index 0c7e82e61..730bf2227 100644 --- a/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md +++ b/roadmaps/inference-engineering/content/mobile-inference@kqzHHoiZ9y4tCS6MMlLHQ.md @@ -1,3 +1,3 @@ # Mobile Inference - + Mobile devices run inference using vendor frameworks: Google's AI Edge SDK for Android and Apple's Foundation Models and Core ML for iOS. Hardware limitations restrict practical model sizes to one to two billion parameters on current phones. Small fine-tuned models for tasks like transcription, translation, and on-device classification are the strongest fit for mobile edge inference. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md b/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md index ec66fcb23..7c966e0f4 100644 --- a/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md +++ b/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md @@ -1,11 +1,10 @@ # Model Evaluation - + Model evaluation, or evals, is the practice of systematically measuring model quality for a specific task or domain. Standard intelligence benchmarks like MMLU measure general capabilities but can be gamed; product-specific evals that test real use cases provide more reliable signal. High-confidence evals are a prerequisite before investing in inference optimization, ensuring the model is actually useful before making it fast. Visit the following resources to learn more: -- [@article@How to Fine-Tune Qwen3 to GPT-4o Level Performance](https://ghost.oxen.ai/how-to-fine-tune-qwen3-to-gpt-4o-level-performance/) - [@article@LLM Evaluation: A Beginner's Guide](https://www.evidentlyai.com/llm-guide/llm-evaluation) - [@article@LLM Evaluation Metrics and Methods, Explained Simply](https://www.evidentlyai.com/llm-guide/llm-evaluation-metrics) -- [@article@A Pragmatic Guide to LLM Evals for Devs](https://newsletter.pragmaticengineer.com/p/evals) -- [@book@Evals for AI Engineers](https://www.oreilly.com/library/view/evals-for-ai/9798341660717/) \ No newline at end of file +- [@article@Automated Regression Testing: The Complete 2026 Guide](https://roadmap.sh/ai-engineer/automated-regression-testing) +- [@video@LLM as a Judge: Scaling AI Evaluation Strategies](https://www.youtube.com/watch?v=trfUBIDeI1Y) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-formats--runtimes@lTjWXB8YRwx11e8lxSgNm.md b/roadmaps/inference-engineering/content/model-formats--runtimes@lTjWXB8YRwx11e8lxSgNm.md index 4bd2ea829..00a62e2dd 100644 --- a/roadmaps/inference-engineering/content/model-formats--runtimes@lTjWXB8YRwx11e8lxSgNm.md +++ b/roadmaps/inference-engineering/content/model-formats--runtimes@lTjWXB8YRwx11e8lxSgNm.md @@ -1,3 +1,5 @@ +# undefined + ### Model Formats and Runtimes - + Before a model can serve requests, its weights must be stored, loaded, and compiled into an executable form. This group covers the file formats used to distribute weights and the runtimes that compile them into optimized inference engines for specific hardware targets. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-parallelism@CiXsdHHL0e2DN4ICI5jeX.md b/roadmaps/inference-engineering/content/model-parallelism@CiXsdHHL0e2DN4ICI5jeX.md index 25f9a2701..78012e620 100644 --- a/roadmaps/inference-engineering/content/model-parallelism@CiXsdHHL0e2DN4ICI5jeX.md +++ b/roadmaps/inference-engineering/content/model-parallelism@CiXsdHHL0e2DN4ICI5jeX.md @@ -1,3 +1,3 @@ # Model Parallelism - + When a model is too large to fit on a single GPU, or when serving latency requires more compute than one device can provide, model parallelism distributes the workload across multiple GPUs or nodes. This group covers the three parallelism strategies and their trade-offs in terms of communication overhead, latency, and hardware requirements. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md b/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md index 260ab4ff6..f9fa781c3 100644 --- a/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md +++ b/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md @@ -1,9 +1,10 @@ # Model Selection - + The most important decision in model performance optimization is which model to use. Smaller models with fewer parameters are faster and cheaper to serve than larger ones. The goal is to find the smallest model that meets quality requirements for the task. Model architecture also determines which inference engines and optimization techniques are available, so popular architectures with broad tooling support are preferable. Visit the following resources to learn more: - [@article@Choosing the Right Model](https://bentoml.com/llm/getting-started/choosing-the-right-model) - [@article@Beyond Vibes: How to Properly Select the Right LLM for the Right Task](https://aws.amazon.com/blogs/machine-learning/beyond-vibes-how-to-properly-select-the-right-llm-for-the-right-task/) -- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) \ No newline at end of file +- [@video@How to Pick the Right AI Foundation Model](https://www.youtube.com/watch?v=pePAAGfh-IU) +- [@video@How to Choose Large Language Models: A Developer’s Guide to LLMs](https://www.youtube.com/watch?v=pYax2rupKEY) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/modelopt@zhiKpdZm8zOg4e3bm1OgN.md b/roadmaps/inference-engineering/content/modelopt@zhiKpdZm8zOg4e3bm1OgN.md index e6bc6e2e6..01edda6fd 100644 --- a/roadmaps/inference-engineering/content/modelopt@zhiKpdZm8zOg4e3bm1OgN.md +++ b/roadmaps/inference-engineering/content/modelopt@zhiKpdZm8zOg4e3bm1OgN.md @@ -1,3 +1,3 @@ # ModelOpt - + ModelOpt is NVIDIA's toolkit for post-training quantization, supporting FP8, INT8, FP4, MXFP8, MXFP4, and NVFP4 output formats. It integrates with PyTorch and exports quantized models directly to TensorRT-LLM, vLLM, and SGLang. Calibration runs a small set of representative inputs through the model to compute per-layer scale factors that minimize the gap between quantized and original outputs. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md b/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md index 0f678708d..9be3b4cb9 100644 --- a/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md +++ b/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md @@ -1,10 +1,10 @@ # Multi-Cloud - + Multi-cloud inference distributes GPU capacity across providers and regions to maximize availability, minimize latency, and meet data residency requirements. This group covers procurement strategy, capacity management, geographic routing, and the reliability patterns that make large-scale inference resilient to hardware and provider failures. Visit the following resources to learn more: -- [@article@SemiAnalysis](https://semianalysis.com/) -- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) - [@book@Designing Data-Intensive Applications](https://dataintensive.net/) -- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file +- [@book@Site Reliability Engineering](https://sre.google/books/) +- [@article@SemiAnalysis](https://semianalysis.com/) +- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/multi-node-inference@FZx2xsqw_cmYLFS3GMAl-.md b/roadmaps/inference-engineering/content/multi-node-inference@FZx2xsqw_cmYLFS3GMAl-.md index df9b6df5c..a219eb083 100644 --- a/roadmaps/inference-engineering/content/multi-node-inference@FZx2xsqw_cmYLFS3GMAl-.md +++ b/roadmaps/inference-engineering/content/multi-node-inference@FZx2xsqw_cmYLFS3GMAl-.md @@ -1,3 +1,3 @@ # Multi-Node Inference - + Multi-node inference spreads a model across more than eight GPUs using InfiniBand for inter-node communication. It is necessary for the largest models where a single node lacks sufficient VRAM for weights and KV cache. InfiniBand bandwidth is an order of magnitude lower than NVLink, so parallelism strategies must minimize cross-node communication. For MoE models, Expert Parallelism is preferred over Tensor Parallelism between nodes. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md b/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md index 3dbc4580e..8edc4eba5 100644 --- a/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md +++ b/roadmaps/inference-engineering/content/n-gram@ZOclQMd7JASGXTRyctfW2.md @@ -1,5 +1,5 @@ # N-gram Speculation - + N-gram speculation builds a dictionary during prefill that maps token prefixes to observed n-gram continuations from the input. During decode, generated tokens are looked up in this dictionary and matching suffixes become draft tokens with no separate model required. It is most effective for code completion, where output frequently mirrors input syntax, and can produce very long draft sequences with high acceptance rates in that domain. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md b/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md index 1df6cd1e0..e96006cee 100644 --- a/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md +++ b/roadmaps/inference-engineering/content/neural-networks@p2S0W3EDn768jbqzc7CE7.md @@ -1,9 +1,10 @@ # Neural Networks - + A neural network is a layered structure of nodes where each node applies a learned weight matrix and bias to its input. Multiple layers connected in sequence form the foundation for all generative AI models. Inference engineers need a working model of how neural networks compute outputs to understand where performance bottlenecks arise. Visit the following resources to learn more: - [@book@Deep Learning](https://www.deeplearningbook.org/) - [@book@Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python-second-edition) -- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) \ No newline at end of file +- [@article@What is deep learning?](https://www.ibm.com/think/topics/deep-learning) +- [@video@But what is a neural network?](https://www.youtube.com/watch?v=aircAruvnKk) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/nims@KB7ExU9X49omOuk9D_Qal.md b/roadmaps/inference-engineering/content/nims@KB7ExU9X49omOuk9D_Qal.md index 2fa8dff0a..e6ef22889 100644 --- a/roadmaps/inference-engineering/content/nims@KB7ExU9X49omOuk9D_Qal.md +++ b/roadmaps/inference-engineering/content/nims@KB7ExU9X49omOuk9D_Qal.md @@ -1,3 +1,3 @@ # NIMs - + NIMs are pre-built Docker containers provided by NVIDIA for popular open models, available as flexible multi-LLM containers or as GPU- and model-specific optimized containers. They provide a ready-made starting point for serving common architectures without building a custom container from scratch. For maximum control and custom configurations, starting from a less opinionated base image and building up is generally preferable. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md b/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md index b873fe678..763e8d5e3 100644 --- a/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md +++ b/roadmaps/inference-engineering/content/nvidia-dynamo@KhTFecXEkHR4WX-_TYY3g.md @@ -1,5 +1,5 @@ # NVIDIA Dynamo - + Dynamo supports conditional disaggregation, where requests are first sent to the decode engine and only routed to a prefill engine when the input is long or not already cached. The ratio of prefill to decode engines is configurable at runtime and can shift as traffic patterns change. This dynamic allocation prevents the prefill queue from becoming a bottleneck and avoids wasting decode engine resources on prefill-heavy traffic. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md b/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md index 7d5dccca1..eff4ee05c 100644 --- a/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md +++ b/roadmaps/inference-engineering/content/nvidia-dynamo@ctdMz6rHa4Cwl1x19V_5k.md @@ -1,5 +1,5 @@ # NVIDIA Dynamo - + Dynamo is NVIDIA's open-source distributed serving platform that sits above inference engines and provides orchestration for large-scale deployments. It adds KV-aware routing, disaggregated prefill/decode serving, multi-node parallelism, and an SLA-based planner that dynamically adjusts the number of prefill and decode workers based on TTFT and TPS targets. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md b/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md index 11d902bb4..825026ed9 100644 --- a/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md +++ b/roadmaps/inference-engineering/content/nvidia@Uq3mE-0WIASHfcBDEnbLj.md @@ -1,3 +1,3 @@ # NVIDIA - + NVIDIA GPUs are the dominant hardware for inference due to their combination of Tensor Core compute, HBM bandwidth, NVLink interconnects, and the CUDA software ecosystem. The three most relevant architectures for current inference deployments are Hopper, Blackwell, and the Grace and Vera CPU platforms that pair with them. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md b/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md index b66f800cb..b7736cac4 100644 --- a/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md +++ b/roadmaps/inference-engineering/content/observability@w-dVv5dFNxD7Rqy16m4uL.md @@ -1,5 +1,5 @@ # Observability - + Inference observability tracks total request volume, input and output sequence lengths, response codes, latency percentiles (P50, P90, P99 of TTFT, TPS, and end-to-end latency), replica count, GPU utilization, and queue depth. These metrics are interdependent; a latency spike could come from long input sequences, high concurrency, or a hardware issue. Observability must integrate with existing tooling like Grafana, Datadog, and PagerDuty to put inference metrics in context. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/omni-modal-models@plw2WiOoGxg64P-Kv-Xm7.md b/roadmaps/inference-engineering/content/omni-modal-models@plw2WiOoGxg64P-Kv-Xm7.md index f1cf34fe6..b27755d02 100644 --- a/roadmaps/inference-engineering/content/omni-modal-models@plw2WiOoGxg64P-Kv-Xm7.md +++ b/roadmaps/inference-engineering/content/omni-modal-models@plw2WiOoGxg64P-Kv-Xm7.md @@ -1,3 +1,3 @@ # Omni-Modal Models - + Omni models accept multiple input modalities and produce multiple output types in a single model. They avoid the latency of running separate specialized models in sequence but often lag behind dedicated models in quality within each modality. Production pipelines using VLMs typically coordinate preprocessing steps like OCR, audio transcription, and PDF extraction as separate independently scaling components. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/online-vs-offline-inf@SkvPauxjmqok4TyNWSACL.md b/roadmaps/inference-engineering/content/online-vs-offline-inf@SkvPauxjmqok4TyNWSACL.md index 9d120a7d3..0015c8e07 100644 --- a/roadmaps/inference-engineering/content/online-vs-offline-inf@SkvPauxjmqok4TyNWSACL.md +++ b/roadmaps/inference-engineering/content/online-vs-offline-inf@SkvPauxjmqok4TyNWSACL.md @@ -1,3 +1,9 @@ # Online vs. Offline Inference - -Online inference serves real-time user-facing requests where latency matters. Offline inference processes large batches of data asynchronously where throughput and cost matter more than per-request speed. The same model, like Whisper for transcription, may need separate deployments for each use case because the optimal configurations differ significantly. \ No newline at end of file + +Online inference serves real-time user-facing requests where latency matters. Offline inference processes large batches of data asynchronously where throughput and cost matter more than per-request speed. The same model, like Whisper for transcription, may need separate deployments for each use case because the optimal configurations differ significantly. + +Visit the following resources to learn more: + +- [@article@Production ML systems: Static versus dynamic inference](https://developers.google.com/machine-learning/crash-course/production-ml-systems/static-vs-dynamic-inference) +- [@article@Trade-offs in Online vs Offline Model Inference](https://bugfree.ai/knowledge-hub/trade-offs-online-vs-offline-model-inference) +- [@video@Offline Recurrence for Improved Online Inference](https://www.youtube.com/watch?v=bwvE1KvLzLQ) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md b/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md index 8c409ae95..00cf1b9fd 100644 --- a/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md +++ b/roadmaps/inference-engineering/content/onnx-runtime@Vmtbdre3SZzwq3FxOO-Bc.md @@ -1,5 +1,5 @@ # ONNX Runtime - + ONNX Runtime is an open-source inference runtime that executes models exported to the ONNX format, which stores both weights and a computation graph for cross-platform portability. It supports multiple hardware backends including CUDA, DirectML, and CPU. It is widely used for embedding models, vision models, and pipelines that need to run across hardware environments without engine-specific configuration. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md b/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md index cddb72aba..195e46487 100644 --- a/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md +++ b/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md @@ -1,9 +1,10 @@ # Open vs. Closed Models - + Closed models are accessed exclusively through a vendor API; the weights are never released. Open models publish their weights for download and self-hosting. For inference engineers, this distinction determines the entire problem space: closed models require no infrastructure but offer no control over latency, cost, or configuration, while open models require building and operating a full inference stack but give complete control over every layer of it. Visit the following resources to learn more: +- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) +- [@article@Open-source Models vs. Closed-Source Models: A Simple Guide](https://koat.ai/blog/open-source-models-vs-closed-source-models-a-simple-guide) - [@article@Open-Source vs. Closed LLMs: A Guide](https://hatchworks.com/blog/gen-ai/open-source-vs-closed-llms-guide/) -- [@video@Open Source vs Closed AI: LLMs, Agents & the AI Stack Explained](https://www.youtube.com/watch?v=_QfxGZGITGw) -- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) \ No newline at end of file +- [@video@Open Source vs Closed AI: LLMs, Agents & the AI Stack Explained](https://www.youtube.com/watch?v=_QfxGZGITGw) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md b/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md index 1ddef2370..6453ece1e 100644 --- a/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md +++ b/roadmaps/inference-engineering/content/other-accelerators@am4Hh-14g6Awk0WcFmVN7.md @@ -1,3 +1,3 @@ # Other Accelerators - + Alternatives to NVIDIA GPUs include AMD MI350, Google TPUs, AWS Inferentia and Trainium, Cerebras WSE-3, and Groq LPUs. Each offers a specific advantage: AMD provides CUDA-compatible tooling via ROCm; TPUs integrate deeply with Google Cloud and JAX; Inferentia and Trainium offer cost advantages within AWS. All face the shared challenge of building an ecosystem to match the breadth of NVIDIA's CUDA libraries and inference engine support. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md b/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md index e95d3254c..bb48f9fd1 100644 --- a/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md +++ b/roadmaps/inference-engineering/content/pagedattention@VcIRY4vSi7JkcV6g13vkN.md @@ -1,5 +1,5 @@ # PagedAttention - + PagedAttention stores the KV cache in fixed-size pages rather than a single contiguous memory block, allowing it to be distributed across GPU memory without fragmentation. It uses a lookup table to access pages on demand, reducing memory waste from over-allocation and enabling more requests to share GPU memory. It is the memory management technique that made vLLM's high throughput possible. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/performance-benchmarking@XXfjlqd0CYHwBR5-Kdmmi.md b/roadmaps/inference-engineering/content/performance-benchmarking@XXfjlqd0CYHwBR5-Kdmmi.md index fa8360259..51496b3b7 100644 --- a/roadmaps/inference-engineering/content/performance-benchmarking@XXfjlqd0CYHwBR5-Kdmmi.md +++ b/roadmaps/inference-engineering/content/performance-benchmarking@XXfjlqd0CYHwBR5-Kdmmi.md @@ -1,3 +1,3 @@ # Performance Benchmarking - + Benchmarking measures inference service performance under realistic workloads. Good benchmarks simulate realistic input and output sequence lengths, concurrency patterns, and request contents rather than synthetic uniform inputs. Standard tools include SGLang Genai-bench, NVIDIA GenAI-Perf, and Locust. Always establish a baseline before applying optimizations and change one variable at a time. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md b/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md index 7ca6319aa..c625450a3 100644 --- a/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md +++ b/roadmaps/inference-engineering/content/pipeline-parallelism@9KLS3gWArBofLEULu3sQc.md @@ -1,5 +1,5 @@ # Pipeline Parallelism - + Pipeline Parallelism splits an LLM's layers into sequential stages and assigns each stage to a different GPU or node. Unlike Tensor Parallelism, each GPU holds complete layers rather than layer fragments. It introduces pipeline bubbles where GPUs wait for preceding stages, reducing efficiency. It is used for multi-node inference with dense models, typically combined with Tensor Parallelism within each node. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/prefill--decode-phases@hwKZmfYFiIu6plbiXtJRI.md b/roadmaps/inference-engineering/content/prefill--decode-phases@hwKZmfYFiIu6plbiXtJRI.md index b81f01f3a..3039f730c 100644 --- a/roadmaps/inference-engineering/content/prefill--decode-phases@hwKZmfYFiIu6plbiXtJRI.md +++ b/roadmaps/inference-engineering/content/prefill--decode-phases@hwKZmfYFiIu6plbiXtJRI.md @@ -1,3 +1,9 @@ # Prefill and Decode Phases - -LLM inference has two distinct phases. Prefill processes the entire input sequence in parallel to build the KV cache and is compute-bound. Decode generates output tokens one at a time by reading model weights from memory and is memory-bound. Each phase has different bottlenecks and responds to different optimization techniques. \ No newline at end of file + +LLM inference has two distinct phases. Prefill processes the entire input sequence in parallel to build the KV cache and is compute-bound. Decode generates output tokens one at a time by reading model weights from memory and is memory-bound. Each phase has different bottlenecks and responds to different optimization techniques. + +Visit the following resources to learn more: + +- [@article@Prefill vs Decode: LLM Inference Phases Explained](https://redis.io/blog/prefill-vs-decode/) +- [@article@Understanding the Two Key Stages of LLM Inference: Prefill and Decode(Part-1)](https://medium.com/@sailakkshmiallada/understanding-the-two-key-stages-of-llm-inference-prefill-and-decode-29ec2b468114) +- [@video@Why LLMs Read Fast but Write Slowly - Prefill vs Decode](https://www.youtube.com/watch?v=hmMgTgQpO38) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md b/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md index efc6f483c..f3b791117 100644 --- a/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md +++ b/roadmaps/inference-engineering/content/profiling-performance@S2GZw9etrrvEJzV9V-1C_.md @@ -1,5 +1,5 @@ # Profiling Performance - + Profiling reveals where time is spent within a benchmark, identifying which operations consume the most compute or memory. Tools include PyTorch Profiler, NVIDIA Nsight Systems for system-wide GPU and CPU analysis, and NVIDIA Nsight Compute for per-kernel analysis. Profiling is most useful when writing custom inference code or contributing to inference frameworks, rather than configuring existing engines. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md b/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md index 0bbb52b09..44610422a 100644 --- a/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md +++ b/roadmaps/inference-engineering/content/pytorch@AHVfxQRXmRoiTxC1eeMme.md @@ -1,5 +1,5 @@ # PyTorch - + PyTorch is the standard Python framework for building and running neural networks. It abstracts CUDA into a tensor operation API and provides automatic differentiation for training. For inference, PyTorch supports torch.compile for automatic kernel optimization and serves as the foundation that all major inference engines build on top of. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md b/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md index 264447515..2607af1e3 100644 --- a/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md +++ b/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md @@ -1,5 +1,5 @@ # Quantization - + Quantization converts model weights and other values from their native precision to a lower-precision format. Cutting precision in half doubles memory bandwidth for decode and enables twice the FLOPS for prefill on compatible Tensor Cores. The tradeoff is potential quality loss from reduced numerical precision, which can compound across the many calculations in a forward pass. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/real-time-factor@0-BRllrMiShLlytidGkST.md b/roadmaps/inference-engineering/content/real-time-factor@0-BRllrMiShLlytidGkST.md index 300980eeb..7ab287f69 100644 --- a/roadmaps/inference-engineering/content/real-time-factor@0-BRllrMiShLlytidGkST.md +++ b/roadmaps/inference-engineering/content/real-time-factor@0-BRllrMiShLlytidGkST.md @@ -1,3 +1,9 @@ # Real-Time Factor (RTF) - -RTF measures how quickly an ASR system can process audio relative to its duration. A system that transcribes one hour of audio in six seconds has an RTF of 600x. It is the standard throughput metric for long-file transcription workloads and improves roughly linearly with the number of GPUs used for parallel chunk processing. \ No newline at end of file + +RTF measures how quickly an ASR system can process audio relative to its duration. A system that transcribes one hour of audio in six seconds has an RTF of 600x. It is the standard throughput metric for long-file transcription workloads and improves roughly linearly with the number of GPUs used for parallel chunk processing. + +Visit the following resources to learn more: + +- [@article@What is RTF (Real-Time Factor) in ASR? Copied from: https://www.futurebeeai.com/knowledge-hub/real-time-factor-asr](https://www.futurebeeai.com/knowledge-hub/real-time-factor-asr) +- [@article@Whisper Real-Time Factor by GPU](https://gigagpu.com/whisper-rtf-by-gpu/) +- [@article@RTF — Real-Time Factor Explained](https://transcript.you/glossary/rtf) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md b/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md index 6f791376d..1e56cba17 100644 --- a/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md +++ b/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md @@ -1,7 +1,10 @@ # Reasoning Models - + Reasoning models generate an intermediate thinking sequence before producing their final output, adding a third token sequence alongside the input and output. This reasoning trace increases total token count significantly and changes inference economics: TTFT is higher, total latency is longer, and output token costs dominate. Inference engineers must account for the variable and sometimes very long reasoning sequences when sizing infrastructure and setting latency budgets. Visit the following resources to learn more: -- [@article@Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) \ No newline at end of file +- [@article@Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) +- [@article@What is a reasoning model?](https://www.ibm.com/think/topics/reasoning-model) +- [@video@What Are Large Reasoning Models (LRMs)? Smarter AI Beyond LLMs](https://www.youtube.com/watch?v=enLbj0igyx4) +- [@video@How do thinking and reasoning models work?](https://www.youtube.com/watch?v=xCRvOUykOX0) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/request-queueing@k-zXOGynI0B0yyT-n3raa.md b/roadmaps/inference-engineering/content/request-queueing@k-zXOGynI0B0yyT-n3raa.md index 07498de5e..73d7f1d29 100644 --- a/roadmaps/inference-engineering/content/request-queueing@k-zXOGynI0B0yyT-n3raa.md +++ b/roadmaps/inference-engineering/content/request-queueing@k-zXOGynI0B0yyT-n3raa.md @@ -1,3 +1,3 @@ # Request Queueing - + A request queue holds incoming traffic when all active replicas are at capacity while autoscaling brings new replicas online. A first-in, first-out queue is the standard implementation; priority queues can give paid users better position. Queues must be visible to newly started replicas so that requests are not held waiting for existing replicas when new capacity becomes available. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md b/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md index 9acda65b9..7daff5637 100644 --- a/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md +++ b/roadmaps/inference-engineering/content/routing--load-balancing@DSyTAtFC1TvC1P7yD10Nb.md @@ -1,9 +1,9 @@ # Routing and Load Balancing - + Routers make request-level decisions about which replica to send a request to, considering factors like KV cache prefix match and LoRA availability. Load balancers make system-level decisions to distribute traffic evenly across replicas. In systems with prefix caching, naive round-robin load balancing misses cache opportunities; intelligent routing increases cache hit rates and reduces TTFT. Visit the following resources to learn more: -- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) - [@book@Designing Data-Intensive Applications](https://dataintensive.net/) -- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file +- [@book@Site Reliability Engineering](https://sre.google/books/) +- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/safetensors@jkbrf0_JDljWmVq_o9XE-.md b/roadmaps/inference-engineering/content/safetensors@jkbrf0_JDljWmVq_o9XE-.md index e2fcd129f..7bc7c1233 100644 --- a/roadmaps/inference-engineering/content/safetensors@jkbrf0_JDljWmVq_o9XE-.md +++ b/roadmaps/inference-engineering/content/safetensors@jkbrf0_JDljWmVq_o9XE-.md @@ -1,3 +1,3 @@ # Safetensors - + Safetensors is the standard format for storing and distributing open model weights. It uses memory mapping for fast, zero-copy loading and prevents execution of arbitrary code during deserialization, making it safer than earlier pickle-based formats. All major inference engines load safetensors natively, and most open models on Hugging Face publish weights in this format. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/scale-to-zero@Pi87dWgu5NTIJRQUc169s.md b/roadmaps/inference-engineering/content/scale-to-zero@Pi87dWgu5NTIJRQUc169s.md index 39d3d1637..ad5cfa4ac 100644 --- a/roadmaps/inference-engineering/content/scale-to-zero@Pi87dWgu5NTIJRQUc169s.md +++ b/roadmaps/inference-engineering/content/scale-to-zero@Pi87dWgu5NTIJRQUc169s.md @@ -1,3 +1,3 @@ # Scale to Zero - + Scale to zero allows the system to run zero replicas during periods of no traffic and spin up on demand. It requires fast cold starts and robust queuing to hold incoming requests while a replica starts. Scale to zero is well-suited for development environments and periodically accessed workloads, but not for latency-sensitive production applications with unpredictable light traffic. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/security-and-compliance@4x-4BFZH7bfNteD5ZAwTI.md b/roadmaps/inference-engineering/content/security-and-compliance@4x-4BFZH7bfNteD5ZAwTI.md index 94cc9df57..68c8a0b4b 100644 --- a/roadmaps/inference-engineering/content/security-and-compliance@4x-4BFZH7bfNteD5ZAwTI.md +++ b/roadmaps/inference-engineering/content/security-and-compliance@4x-4BFZH7bfNteD5ZAwTI.md @@ -1,3 +1,3 @@ # Security and Compliance - + Inference infrastructure must protect user inputs, model outputs, and model weights. Standard practices include data encryption, container security, network isolation, and access controls validated by third-party penetration testing. Regulated industries require compliance with frameworks like SOC 2 and HIPAA, which also apply to underlying cloud providers. Multi-cloud deployments support data residency requirements by routing each country's data to clusters in that country. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md b/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md index 2bb16ac62..c7e64b9b9 100644 --- a/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md +++ b/roadmaps/inference-engineering/content/sglang@YtlAf6qloKMmaSr9Bkkl_.md @@ -1,5 +1,5 @@ # SGLang - + SGLang is a high-throughput inference engine with a flexible frontend language for customizing server behavior. It has strong support for MoE models like DeepSeek and Qwen and works closely with model labs to optimize new architectural features. SGLang Diffusion extends it to image and video generation models with pipeline abstractions and diffusion-specific parallelism. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md b/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md index ae95e3991..8c90d1204 100644 --- a/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md +++ b/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md @@ -1,8 +1,10 @@ # Shared vs. Dedicated Inference - + Shared inference means sending requests to a public API and paying per token. Dedicated inference means renting GPUs and running your own model server, paying per GPU-hour. Shared inference is simpler and has no cold start, but dedicated inference offers control over latency, cost at scale, model customization, and uptime. Most products start shared and move to dedicated as traffic grows. Visit the following resources to learn more: +- [@article@Dedicated vs shared LLM inference: when reserved capacity makes sense](https://www.freestyle.sh/blog/engineering/dedicated-vs-shared-llm-inference) - [@article@Serverless vs. Self-Hosted LLM Inference](https://bentoml.com/llm/llm-inference-basics/serverless-vs-self-hosted-llm-inference) -- [@article@Self-Hosted LLM: A Practical Guide for DevOps](https://www.plural.sh/blog/self-hosting-large-language-models/) \ No newline at end of file +- [@article@Self-Hosted LLM: A Practical Guide for DevOps](https://www.plural.sh/blog/self-hosting-large-language-models/) +- [@article@When shared inference is not enough: the case for dedicated AI endpoints](https://www.gmicloud.ai/en/blog/when-shared-inference-is-not-enough-the-case-for-dedicated-ai-endpoints) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md b/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md index 74e7213cd..544e58b69 100644 --- a/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md +++ b/roadmaps/inference-engineering/content/sizing--procurement@da0HGfSDr6_DV4ZJcrcIB.md @@ -1,9 +1,9 @@ # Sizing and Procurement - + Estimating GPU requirements starts with VRAM: roughly 1 GB per billion parameters at FP8 for weights, plus at least 50 percent headroom for the KV cache. This total is rounded up to the nearest available instance size. Cloud GPUs are available from hyperscalers like AWS and GCP, neoclouds like CoreWeave and Nebius, and spot markets; large deployments combine reserved instances for baseline load with on-demand and spot capacity for bursts. Visit the following resources to learn more: -- [@article@SemiAnalysis](https://semianalysis.com/) - [@book@Designing Data-Intensive Applications](https://dataintensive.net/) -- [@book@Site Reliability Engineering](https://sre.google/books/) \ No newline at end of file +- [@book@Site Reliability Engineering](https://sre.google/books/) +- [@article@SemiAnalysis](https://semianalysis.com/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md b/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md index 0e185c989..b2e0aed08 100644 --- a/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md +++ b/roadmaps/inference-engineering/content/speculative-decoding@gMWgd10cxcQG_RG2oSAEH.md @@ -1,5 +1,5 @@ # Speculative Decoding - + Speculative decoding takes advantage of spare compute during memory-bound decode by generating multiple draft tokens and validating them in a single forward pass. If the target model accepts N draft tokens, the forward pass produces N+1 tokens instead of one. The performance gain depends on draft token cost, draft sequence length, and acceptance rate, which decreases at higher temperatures and larger batch sizes. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/speech@Lk8906Mpi3yxiSt8EC_BC.md b/roadmaps/inference-engineering/content/speech@Lk8906Mpi3yxiSt8EC_BC.md index a90d79f52..8718cd12d 100644 --- a/roadmaps/inference-engineering/content/speech@Lk8906Mpi3yxiSt8EC_BC.md +++ b/roadmaps/inference-engineering/content/speech@Lk8906Mpi3yxiSt8EC_BC.md @@ -1,3 +1,3 @@ # Speech - + Speech inference covers the pipeline from raw audio to text and back, including the detection, transcription, diarization, and synthesis steps that make up voice applications. Each component has distinct inference characteristics and often runs on separate hardware to avoid resource contention. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md b/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md index 097129910..a6b94eb39 100644 --- a/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md +++ b/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md @@ -1,5 +1,5 @@ # Streaming and Protocol Support - + Text streaming over HTTP is sufficient for LLM chat applications. For real-time voice and video applications, WebSockets provide a persistent bidirectional connection for continuous unstructured data like audio chunks. gRPC provides bidirectional streaming for structured service-to-service communication with schema enforcement. Each WebSocket connection holds a slot on the server, and the concurrency limit must be set to match the GPU's sustainable load. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/streaming-tts@3VbPNaSnzWZ-pC-lGFMrh.md b/roadmaps/inference-engineering/content/streaming-tts@3VbPNaSnzWZ-pC-lGFMrh.md index 503a436d2..e6d4a4206 100644 --- a/roadmaps/inference-engineering/content/streaming-tts@3VbPNaSnzWZ-pC-lGFMrh.md +++ b/roadmaps/inference-engineering/content/streaming-tts@3VbPNaSnzWZ-pC-lGFMrh.md @@ -1,3 +1,3 @@ # Streaming TTS - + Real-time TTS streams audio continuously over WebSockets rather than returning a complete file after generation finishes. The audio decoder converts token streams to waveforms as they are generated, implemented in compiled PyTorch with dynamic batching. After tuning the inference engine and decoder, the primary performance lever is how many concurrent real-time streams a single GPU replica can sustain. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md b/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md index 01e480138..6a21c2d40 100644 --- a/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md +++ b/roadmaps/inference-engineering/content/tensor-parallelism@csghcNLDywO8oTIZNXeyP.md @@ -1,5 +1,5 @@ # Tensor Parallelism - + Tensor Parallelism splits each layer's weight matrices across multiple GPUs and distributes the matmul computation. Each GPU handles a shard of each layer and the results are combined via an all-reduce operation before the next layer. It reduces per-user latency by sharing both VRAM and compute across GPUs, but requires fast NVLink interconnects and is not well-suited for multi-node deployments. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md b/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md index 88c0bba93..206275b4b 100644 --- a/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md +++ b/roadmaps/inference-engineering/content/tensorrt-llm@n2_zIGkQgyYCMYPypz3vb.md @@ -1,5 +1,5 @@ # TensorRT-LLM - + TensorRT-LLM is NVIDIA's inference engine combining PyTorch-based model execution with proprietary NVIDIA kernels unavailable in other engines. It achieves the highest performance among the three major engines, with excellent support for Hopper and Blackwell architectures and NVIDIA-specific formats like NVFP4. The tradeoff is a steeper learning curve and narrower model support compared to vLLM or SGLang. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md b/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md index f8dcc59d4..7b1c327c0 100644 --- a/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md +++ b/roadmaps/inference-engineering/content/tensorrt@M4qjJSEAnsjrmP3A_O8HC.md @@ -1,5 +1,5 @@ # TensorRT - + TensorRT is NVIDIA's high-performance inference runtime that compiles PyTorch or ONNX models into optimized engines using fused kernels, quantization, and hardware-specific tuning. It is the standard choice for image and video generation models and for any workload where maximum throughput on NVIDIA hardware is the primary goal. Compiled engines are cached to disk, eliminating recompilation overhead on subsequent cold starts. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/testing--deployment@ZlOtQtxJST10nuHjhbUx6.md b/roadmaps/inference-engineering/content/testing--deployment@ZlOtQtxJST10nuHjhbUx6.md index f1d03943c..793c71019 100644 --- a/roadmaps/inference-engineering/content/testing--deployment@ZlOtQtxJST10nuHjhbUx6.md +++ b/roadmaps/inference-engineering/content/testing--deployment@ZlOtQtxJST10nuHjhbUx6.md @@ -1,3 +1,3 @@ # Testing and Deployment - + Deploying inference services safely requires tools for rolling out changes without downtime, estimating costs before committing to a configuration, and maintaining visibility into system health once live. This group covers the deployment, cost, observability, and compliance concerns that apply once a service is ready for production traffic. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/text-to-speech-tts@XnhhN1QVWzlp9DUwkii9-.md b/roadmaps/inference-engineering/content/text-to-speech-tts@XnhhN1QVWzlp9DUwkii9-.md index ba63358ab..33cdc9630 100644 --- a/roadmaps/inference-engineering/content/text-to-speech-tts@XnhhN1QVWzlp9DUwkii9-.md +++ b/roadmaps/inference-engineering/content/text-to-speech-tts@XnhhN1QVWzlp9DUwkii9-.md @@ -1,3 +1,3 @@ # Text-to-Speech (TTS) - + Modern TTS models like Orpheus TTS are fine-tuned LLMs with an expanded vocabulary that includes audio tokens. They generate audio token sequences decoded to waveforms by a small audio decoder. The key performance metrics are time to first byte of audio (TTFB) and the number of concurrent real-time streams a single GPU replica can sustain, rather than raw token throughput. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/the-inference-stack@uIhVVlMsykuxJCa02f-i-.md b/roadmaps/inference-engineering/content/the-inference-stack@uIhVVlMsykuxJCa02f-i-.md index 1f8bfb209..d8e27cbf8 100644 --- a/roadmaps/inference-engineering/content/the-inference-stack@uIhVVlMsykuxJCa02f-i-.md +++ b/roadmaps/inference-engineering/content/the-inference-stack@uIhVVlMsykuxJCa02f-i-.md @@ -1,3 +1,3 @@ # The Inference Stack - + A complete inference system has three layers: runtime, infrastructure, and tooling. The runtime layer optimizes a single model on a single instance using techniques like batching, caching, quantization, and speculation. The infrastructure layer scales across clusters, regions, and cloud providers while maintaining uptime. The tooling layer provides the right level of abstraction for engineers to control the system productively. All three must work together for mission-critical inference at scale. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/time-to-first-token@xCgNhwbS_ELxEyRHD1enO.md b/roadmaps/inference-engineering/content/time-to-first-token@xCgNhwbS_ELxEyRHD1enO.md index 261aa28b3..9488e3324 100644 --- a/roadmaps/inference-engineering/content/time-to-first-token@xCgNhwbS_ELxEyRHD1enO.md +++ b/roadmaps/inference-engineering/content/time-to-first-token@xCgNhwbS_ELxEyRHD1enO.md @@ -1,3 +1,9 @@ # Time to First Token (TTFT) - -TTFT measures how long it takes for a user to receive the first output token after submitting a request. It is determined primarily by the prefill phase and by queue time. For streaming applications like chatbots, TTFT is the most visible latency signal and directly affects perceived responsiveness. \ No newline at end of file + +TTFT measures how long it takes for a user to receive the first output token after submitting a request. It is determined primarily by the prefill phase and by queue time. For streaming applications like chatbots, TTFT is the most visible latency signal and directly affects perceived responsiveness. + +Visit the following resources to learn more: + +- [@article@What is time to first token (TTFT)?](https://www.ibm.com/think/topics/time-to-first-token) +- [@article@What Is Time to First Token?](https://www.cerebras.ai/glossary/what-is-time-to-first-token) +- [@video@Most devs don't understand how LLM tokens work](https://www.youtube.com/watch?v=nKSk_TiR8YA) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/tokens-per-second@uxm_BIAyhfFmcE94k9i9J.md b/roadmaps/inference-engineering/content/tokens-per-second@uxm_BIAyhfFmcE94k9i9J.md index 97ddc562d..1c85689fe 100644 --- a/roadmaps/inference-engineering/content/tokens-per-second@uxm_BIAyhfFmcE94k9i9J.md +++ b/roadmaps/inference-engineering/content/tokens-per-second@uxm_BIAyhfFmcE94k9i9J.md @@ -1,3 +1,9 @@ # Tokens Per Second (TPS) - -TPS measures how many tokens are delivered to the user per second after the first token. It is determined by the decode phase and reflects memory bandwidth efficiency. Per-user TPS (also called perceived TPS) is a latency metric; total TPS across the inference service is a throughput metric. \ No newline at end of file + +TPS measures how many tokens are delivered to the user per second after the first token. It is determined by the decode phase and reflects memory bandwidth efficiency. Per-user TPS (also called perceived TPS) is a latency metric; total TPS across the inference service is a throughput metric. + +Visit the following resources to learn more: + +- [@article@What Are Tokens per Second?](https://www.cerebras.ai/glossary/what-are-tokens-per-second) +- [@article@Tokens Per Second](https://www.taskade.com/wiki/ai/tokens-per-second) +- [@video@LLM Speed Tiers Explained: What TPS (Token Per Second) Actually Feels Like](https://www.youtube.com/watch?v=EZE-2xYpvOw) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md b/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md index 097af156b..f9e2bd2d7 100644 --- a/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md +++ b/roadmaps/inference-engineering/content/transformer-blocks@xHM7qjigMmXEwxeMEeMWl.md @@ -1,5 +1,5 @@ # Transformer Blocks - + Transformer blocks are the repeated hidden layers of an LLM. Each block contains an attention sublayer, a feed-forward neural network sublayer, and normalization layers. The feed-forward network holds the majority of model parameters, while attention is the most computationally complex operation per token. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md b/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md index 73b0c1a1e..47f53e9ab 100644 --- a/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md +++ b/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md @@ -1,9 +1,9 @@ # Hugging Face Transformers and Diffusers - + The transformers and diffusers libraries provide reference implementations of popular LLM and image/video generation architectures. They are not designed for production serving but are essential for understanding model configurations, downloading weights, and prototyping inference. Model cards on Hugging Face use transformers or diffusers sample code as the canonical usage examples. Visit the following resources to learn more: -- [@official@Hugging Face Diffusers](https://huggingface.co/docs/diffusers/index) -- [@official@Hugging Face Transformers](https://huggingface.co/docs/transformers/index) - [@course@Hugging Face Course](https://www.youtube.com/playlist?list=PLo2EIpI_JMQvWfQndUesu0nPBAtZ9gP1o) +- [@official@Hugging Face Diffusers](https://huggingface.co/docs/diffusers/index) +- [@official@Hugging Face Transformers](https://huggingface.co/docs/transformers/index) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md b/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md index e6b6bda00..5140d8a74 100644 --- a/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md +++ b/roadmaps/inference-engineering/content/video-generation-inference@PqEH1ez5dg5tEsspgfAK1.md @@ -1,5 +1,5 @@ # Video Generation Inference - + Video generation models run on full nodes of eight GPUs with a batch size of one because the latent space is too large to batch effectively. Attention over the three-dimensional video latent space consumes 70 to 80 percent of compute time and is the primary optimization target. Attention caching techniques that reuse outputs from previous timesteps or transformer layers can improve speed by 30 to 40 percent. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md b/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md index bb589aa69..b562614ff 100644 --- a/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md +++ b/roadmaps/inference-engineering/content/video-generation-models@yTSrQfJsE-Mf4ZU_hJwYT.md @@ -1,5 +1,5 @@ # Video Generation Models - + Video generation models extend image generation to three-dimensional latent space, representing width, height, and time simultaneously. They run the same number of denoising steps as image models but process far more data per step. Attention over the full video latent space dominates compute time, making video generation the most resource-intensive common inference workload. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md b/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md index 54ea63650..bf301ff84 100644 --- a/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md +++ b/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md @@ -1,5 +1,5 @@ # Video Processing for VLMs - + One second of video contains 24 frames, each requiring roughly 1,000 tokens to represent at full resolution. This makes video inputs extremely long-context workloads. Frame rate reduction and downsampling are practically mandatory for running video through a VLM in a single request. Prefix caching and chunked prefill become critical for managing the resulting multi-thousand-token sequences. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md b/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md index 6c36e729b..422979952 100644 --- a/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md +++ b/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md @@ -1,11 +1,11 @@ # Vision Language Models (VLMs) - + VLMs combine a large language model with a small vision encoder that converts input images or video into a sequence of visual tokens. A high-resolution image typically adds about 1,000 tokens to the input sequence. All LLM inference optimization techniques apply to VLMs, but longer sequences make quantization, prefix caching, and disaggregation especially important. Visit the following resources to learn more: +- [@course@A Multimodal World - Hugging Face Computer Vision Course](https://huggingface.co/learn/computer-vision-course/en/unit4/multimodal-models/a_multimodal_world) - [@article@BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597) - [@article@Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) - [@article@Segment Anything](https://arxiv.org/abs/2304.02643) -- [@article@SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://arxiv.org/abs/2509.11815) -- [@course@A Multimodal World - Hugging Face Computer Vision Course](https://huggingface.co/learn/computer-vision-course/en/unit4/multimodal-models/a_multimodal_world) +- [@article@SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://arxiv.org/abs/2509.11815) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md b/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md index 934a0d996..fc34dd700 100644 --- a/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md +++ b/roadmaps/inference-engineering/content/vllm@wcAtbab81IJdpFh6L-heP.md @@ -1,5 +1,5 @@ # vLLM - + vLLM is the most widely adopted LLM inference engine, offering broad support for model architectures, hardware platforms (NVIDIA, AMD, Google TPUs), and optimization techniques including prefix caching, quantization, speculative decoding, and parallelism. Its primary advantage is ease of adoption and the fastest day-zero support for newly released open models. It also supports multimodal inference via vLLM Omni. Visit the following resources to learn more: diff --git a/roadmaps/inference-engineering/content/voice-activity-detection@P0BA_p3xqpxkQsFHqM0AZ.md b/roadmaps/inference-engineering/content/voice-activity-detection@P0BA_p3xqpxkQsFHqM0AZ.md index 1e8542f94..ab8cfa975 100644 --- a/roadmaps/inference-engineering/content/voice-activity-detection@P0BA_p3xqpxkQsFHqM0AZ.md +++ b/roadmaps/inference-engineering/content/voice-activity-detection@P0BA_p3xqpxkQsFHqM0AZ.md @@ -1,3 +1,3 @@ # Voice Activity Detection (VAD) - + VAD is a lightweight model that monitors an audio stream or file and identifies segments containing speech, filtering out silence. It is a required preprocessing component in both real-time ASR pipelines and long-file transcription pipelines, splitting audio into discrete chunks at natural speech boundaries rather than fixed time intervals. VAD runs on dedicated hardware to avoid competing with the ASR model for GPU resources. \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md b/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md index 1a3694d14..d919cfd2b 100644 --- a/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md +++ b/roadmaps/inference-engineering/content/writing-custom-kernels@1dWPVaSQ4TfVU3Obx3m19.md @@ -1,12 +1,12 @@ # Writing Custom Kernels - + Custom kernels are written when existing libraries do not fully utilize a GPU's capabilities for a specific operation or architecture. Most production kernels are written in C++ using CUTLASS and CuTe for GEMM-style operations and FlashInfer for attention variants; Triton offers a Python-based alternative with a lower barrier to entry. Writing an effective kernel requires understanding the target GPU's memory hierarchy, SM occupancy, and Tensor Core instruction set — skills that sit at the intersection of software and hardware. Visit the following resources to learn more: +- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example) - [@official@CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/) - [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/) - [@opensource@NVIDIA/cutlass](https://github.com/NVIDIA/cutlass) - [@opensource@deepseek-ai/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM) -- [@article@FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://arxiv.org/abs/2501.01005) -- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example) \ No newline at end of file +- [@article@FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://arxiv.org/abs/2501.01005) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/zero-downtime-deplo@le4sPQ0h9l-vlmCB4G8ZY.md b/roadmaps/inference-engineering/content/zero-downtime-deplo@le4sPQ0h9l-vlmCB4G8ZY.md index 9a9aef8e0..268b9db50 100644 --- a/roadmaps/inference-engineering/content/zero-downtime-deplo@le4sPQ0h9l-vlmCB4G8ZY.md +++ b/roadmaps/inference-engineering/content/zero-downtime-deplo@le4sPQ0h9l-vlmCB4G8ZY.md @@ -1,3 +1,3 @@ # Zero-Downtime Deployment - + Canary deployments update inference services without interrupting live traffic by routing a small percentage of requests to the new deployment, monitoring for issues, and gradually shifting traffic over. Blue-green deployments, which cut all traffic at once to a parallel environment, are impractical for large inference workloads because they require doubling GPU capacity during the transition. With autoscaling, canary deployments are low cost since the old service scales down as the new one takes traffic. \ No newline at end of file