chore: sync content to repo (#10410)

Co-authored-by: nilbuild <4921183+nilbuild@users.noreply.github.com>
2026-10-24 01:08:07 +08:00 · 2026-10-01 16:20:36 +02:00
parent 9d1aca1463
commit b62fe07c77
125 changed files with 272 additions and 177 deletions
@@ -1,3 +1,10 @@
 # AI-Native Applications
- 
-AI-native applications are products where generative models power the core user experience, from code editors and voice agents to customer support and media generation. Each category has different inference requirements. Agents generate many inference calls per user action; chat applications prioritize TTFT; voice applications need low end-to-end latency for natural conversation.
+
+AI-native applications are products where generative models power the core user experience, from code editors and voice agents to customer support and media generation. Each category has different inference requirements. Agents generate many inference calls per user action; chat applications prioritize TTFT; voice applications need low end-to-end latency for natural conversation.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit the Dedicated AI Product Builder Roadmap](https://roadmap.sh/ai-product-builder)
+- [@article@What is AI native?](https://www.ibm.com/think/topics/ai-native)
+- [@article@Guide to Building AI-Native Applications](https://talkdev.com/learning-center/guide-to-building-ai-native-applications)
+- [@video@How to Build an AI-Native Services Company](https://www.youtube.com/watch?v=gSNFJbgoaHI)
@@ -1,3 +1,8 @@
 # Asynchronous Inference
- 
-Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing.
+
+Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing.
+
+Visit the following resources to learn more:
+
+- [@article@Using asynchronous inference in production](https://www.baseten.co/blog/using-asynchronous-inference-in-production/)
+- [@article@Asynchronous Inference: Methods & Applications](https://www.emergentmind.com/topics/asynchronous-inference)
@@ -1,3 +1,3 @@
 # Asynchronous Inference
- 
+
 Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing.
@@ -1,5 +1,5 @@
 # Attention Mechanism
- 
+
 Attention is the mechanism transformers use to relate each token to all previous tokens via query, key, and value projections. During decode, each new token attends to every prior token, and the KV cache stores precomputed keys and values to avoid recomputation. Attention is the most expensive and most heavily optimized operation in LLM inference.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # Attention Variants
- 
+
 Newer attention algorithms reduce the quadratic time complexity of standard attention. Sliding window attention limits each token to attending to the nearest w tokens, turning O(N²) into O(Nw). Linear attention approximates the softmax with a linear-time kernel. Compressed attention periodically compresses earlier context. These variants trade off some quality for better scaling on long sequences.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # Automatic Speech Recognition (ASR)
- 
+
 ASR models like Whisper take audio input and produce text transcripts. Whisper is an encoder-decoder model where the decoder, an autoregressive transformer, dominates inference time. TensorRT-LLM with in-flight batching for the decoder delivers the best performance. For real-time transcription, streaming via WebSockets with VAD-segmented chunks achieves round-trip targets of around 200 milliseconds per chunk on optimized deployments.

 Visit the following resources to learn more:
@@ -1,9 +1,9 @@
 # Autoscaling
- 
+
 Autoscaling dynamically adjusts the number of active model replicas to match incoming traffic, maintaining latency SLAs while minimizing idle GPU spend. Kubernetes is the standard orchestration layer, scaling replicas within a cluster based on traffic signals and utilization metrics. Key configuration parameters are min replicas, max replicas, the autoscaling window, the scale-down delay, and the concurrency target per replica.

 Visit the following resources to learn more:

- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
 - [@book@Designing Data-Intensive Applications](https://dataintensive.net/)
- [@book@Site Reliability Engineering](https://sre.google/books/)
+- [@book@Site Reliability Engineering](https://sre.google/books/)
+- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
@@ -1,3 +1,3 @@
 # Benchmarking and Profiling
- 
+
 Benchmarking and profiling are how inference engineers turn intuition into evidence. Benchmarking measures system-level throughput and latency under load; profiling zooms in to identify which specific operations are consuming time. Both are prerequisites for making optimization decisions with confidence.
@@ -1,5 +1,5 @@
 # Blackwell
- 
+
 The Blackwell architecture, released in late 2024 with the B200 and B300, added FP4 Tensor Cores and new microscaling number formats including MXFP8, MXFP4, and NVFP4. B200 GPUs offer approximately 5 petaFLOPS of FP8 compute and up to 8 TB/s of memory bandwidth. They are the current gold standard for demanding inference workloads including large LLMs and video generation.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Cache-Aware Routing
- 
+
 In deployments with multiple replicas, cache-aware routing directs requests to the replica most likely to have a matching prefix already in its KV cache. Without this, a user in a long conversation may hit different replicas each turn, getting cache misses that increase latency. A global KV cache stored on networked SSD allows all replicas to eventually access any cached sequence.
@@ -1,5 +1,5 @@
 # Caching Approaches
- 
+
 KV cache reuse is the primary caching strategy in LLM inference, covering both prefix caching (reusing cached KV values across requests that share a common prefix) and KV cache offloading (moving less-used cache blocks to slower storage tiers to free GPU VRAM). A related concern is cache-aware routing, which directs requests to replicas most likely to already hold the relevant cached prefix.

 Visit the following resources to learn more:
@@ -7,4 +7,4 @@ Visit the following resources to learn more:
 - [@opensource@LMCache](https://github.com/LMCache/LMCache)
 - [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444)
 - [@article@What is Prompt Caching?](https://www.ibm.com/think/topics/prompt-caching)
- [@video@What is Prompt Caching? Optimize LLM Latency with AI Transformers](https://www.youtube.com/watch?v=u57EnkQaUTY)
+- [@video@What is Prompt Caching? Optimize LLM Latency with AI Transformers](https://www.youtube.com/watch?v=u57EnkQaUTY)
@@ -1,3 +1,3 @@
 # Caching
- 
+
 Caching reuses previously computed KV values to avoid redundant prefill computation across requests that share common context. It is one of the most impactful techniques for reducing TTFT in workloads with long system prompts or multi-turn conversations. This group covers the approaches, storage tiers, routing implications, and long-context strategies that make caching effective at scale.
@@ -1,3 +1,3 @@
 # Capacity Management
- 
+
 Multi-cloud inference pools GPU capacity from multiple cloud providers and regions into a single schedulable resource, treating distinct compute pools as fungible. This requires a global control plane for deployment and scaling decisions alongside per-cluster workload planes that serve traffic and report utilization. True multi-cloud unlocks greater total capacity, improved reliability through redundancy, lower latency via geographic proximity, and data sovereignty compliance.
@@ -1,5 +1,5 @@
 # Client Integration
- 
+
 The client layer connects application code to the inference service and contributes its own slice of end-to-end latency. This group covers the connection overhead, async request patterns, and protocol choices that determine how efficiently client and server communicate.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Client Latency Overhead
- 
+
 Establishing a new connection between a client and inference server takes tens of milliseconds for TLS handshake, which can represent ten percent or more of a tight latency budget. Session reuse across requests eliminates this overhead for returning clients. Standard SDKs like the OpenAI SDK handle session reuse automatically; custom clients must implement it explicitly.
@@ -1,3 +1,3 @@
 # Cold Starts
- 
+
 A cold start is the time from zero replicas to serving the first request, including GPU provisioning, container image loading, model weight loading, and inference engine startup or compilation. Reducing cold start time is essential for autoscaling systems to scale down confidently and scale up responsively. Key techniques include quantized weights, local weight caching near the GPU instances, and caching compiled inference engines to disk.
@@ -1,3 +1,3 @@
 # Compute
- 
+
 A GPU's compute capability is determined by its Streaming Multiprocessors (SMs), each containing CUDA Cores for general arithmetic and Tensor Cores for matrix multiply-accumulate operations. Tensor Cores are the primary resource for inference, executing the matmuls that make up the majority of forward pass computation. GPU compute is measured in peak Tensor Core FLOPS at a given precision, and roughly doubles with each new NVIDIA architecture generation.
@@ -1,3 +1,7 @@
 # Consumer vs. B2B Workloads
- 
-Consumer applications are cost-sensitive and have unpredictable traffic spikes, requiring flexible autoscaling and low marginal cost. Business-to-business applications prioritize uptime and consistent low latency, as they often sit in the critical path of revenue-generating workflows. Compliance requirements, including data residency and regulatory certifications, are more common in enterprise deployments.
+
+Consumer applications are cost-sensitive and have unpredictable traffic spikes, requiring flexible autoscaling and low marginal cost. Business-to-business applications prioritize uptime and consistent low latency, as they often sit in the critical path of revenue-generating workflows. Compliance requirements, including data residency and regulatory certifications, are more common in enterprise deployments.
+
+Visit the following resources to learn more:
+
+- [@article@Consumer vs. B2B](https://www.baseten.co/inference-engineering/book/01-prerequisites/about-your-app/#consumer-versus-b2b)
@@ -1,7 +1,10 @@
 # Containerization
- 
+
 Containerization packages an inference service and all its dependencies into a Docker image that runs consistently across environments. Inference containers include the CUDA toolkit, Python packages, the inference engine, and any system libraries like ffmpeg for audio processing. Using a pinned dependency tree prevents breaking changes from affecting running services and ensures reproducible builds.

 Visit the following resources to learn more:

- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/)
+- [@official@Docker Documentation](https://docs.docker.com/)
+- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/)
+- [@article@What is containerization?](https://www.ibm.com/think/topics/containerization)
+- [@video@Containerization Explained](https://www.youtube.com/watch?v=0qotVMX-J5s)
@@ -1,3 +1,3 @@
 # Continuous Batching
- 
+
 Continuous batching operates at the token level, swapping requests in and out of the batch as they complete individual tokens rather than waiting for a full batch to finish. This keeps GPU utilization high and minimizes the latency penalty of batching compared to static or dynamic batching. All major inference engines implement continuous batching by default.
@@ -1,5 +1,5 @@
 # Cost Estimation
- 
+
 Dedicated inference cost depends on batch sizing, traffic patterns, and sequence lengths rather than simply on token count. Comparing dedicated versus API costs requires converting per-token API pricing into total cost over a representative time window and comparing it to GPU-hour costs at the same traffic volume. Total cost of ownership also includes engineering time for building and maintaining the inference system.

 Visit the following resources to learn more:
@@ -1,9 +1,9 @@
 # CUDA
- 
+
 CUDA is NVIDIA's proprietary platform for programming GPUs, consisting of a kernel language (C++), a driver for hardware management, and a runtime API for launching kernels. Every inference engine, framework, and library in the NVIDIA ecosystem depends on CUDA. Understanding CUDA is the foundation for diagnosing performance problems that tools and engines cannot solve automatically.

 Visit the following resources to learn more:

+- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example)
 - [@official@CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/)
- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/)
- [@book@CUDA by Example: An Introduction to General-Purpose GPU Programming](https://developer.nvidia.com/cuda-example)
+- [@official@cuBLAS Documentation](https://docs.nvidia.com/cuda/cublas/)
@@ -1,5 +1,5 @@
 # Dependency Management
- 
+
 Inference dependency chains are long and fragile due to the tight coupling between CUDA versions, framework versions, and hardware architectures. Pinning exact versions of every dependency prevents unexpected breakage when upstream packages release updates. Containers for newly released models often initially depend on pre-release software builds and need to be rebuilt on stable releases within days of the model launch.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # Desktop Inference
- 
+
 Desktop inference runs on consumer hardware, primarily Apple M-series unified memory systems and high-end NVIDIA consumer GPUs. Apple hardware offers very large unified memory capacity at moderate bandwidth, enabling larger models than discrete consumer GPUs; NVIDIA consumer GPUs offer higher bandwidth with smaller capacity. Tools like Ollama and llama.cpp make it practical to run aggressively quantized models on consumer hardware with minimal setup.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # Diffusion Models
- 
+
 Diffusion models generate images by starting from random noise and iteratively denoising it over 30 to 50 steps, guided by a text prompt. Each step runs the denoiser twice, once with conditioning and once without, to apply classifier-free guidance. The entire process operates in a low-dimensional latent space rather than full pixel space to make attention computationally feasible.

 Visit the following resources to learn more:
@@ -9,4 +9,4 @@ Visit the following resources to learn more:
 - [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952)
 - [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748)
 - [@article@How DALL-E 2 Actually Works](https://www.assemblyai.com/blog/how-dall-e-2-actually-works/)
- [@video@How AI Image Generators Work (Stable Diffusion / DALL-E)](https://www.youtube.com/watch?v=1CIpzeNxIhU)
+- [@video@How AI Image Generators Work (Stable Diffusion / DALL-E)](https://www.youtube.com/watch?v=1CIpzeNxIhU)
@@ -1,5 +1,5 @@
 # Disaggregation
- 
+
 Disaggregation separates the prefill and decode phases onto independent GPU instances that can scale separately. The prefill engine processes the input sequence and sends the resulting KV cache to the decode engine via the hardware interconnect. This removes the resource contention between compute-heavy prefill and memory-heavy decode. Disaggregation is most effective at high traffic volumes, for large models, and with prefill-heavy workloads like code editors.

 Visit the following resources to learn more:
@@ -1,7 +1,11 @@
 # Distillation
- 
+
 Distillation trains a smaller student model to emulate a larger teacher model by exposing it to the teacher's probability distributions, not just its final outputs. Unlike fine-tuning on synthetic data, distillation transfers the teacher's reasoning behavior. The resulting student model is smaller and faster while retaining much of the teacher's quality on the same tasks.

 Visit the following resources to learn more:

- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042)
+- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042)
+- [@article@How Knowledge Distillation Works and When to Use It](https://www.arcee.ai/blog/how-knowledge-distillation-works-and-when-to-use-it)
+- [@article@Introduction to model distillation: Efficient knowledge transfer for AI applications](https://nebius.com/blog/posts/model-distillation-intro)
+- [@video@Model Distillation: How to Steal a Billion-Dollar AI](https://www.youtube.com/watch?v=QBG6bXiOq40)
+- [@video@Knowledge Distillation - Explained](https://www.youtube.com/watch?v=Pwo1bLDB-jA)
@@ -1,5 +1,5 @@
 # Draft-Target Speculative Decoding
- 
+
 Draft-target speculation uses a small draft model, typically at least ten times smaller than the target, to generate draft tokens for the target model to verify. The draft model should share the same tokenizer and general behavior as the target, often being a smaller member of the same model family. It is the simplest speculation method to set up but introduces the most overhead due to running a separate model.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # EAGLE
- 
+
 EAGLE is a purpose-built speculation method where a small draft model is trained to accept hidden states from the target model as input. By seeing the target's internal representations, EAGLE achieves higher acceptance rates and longer draft sequences (up to eight tokens) than using a general-purpose small model. It integrates into the same forward pass as the target model, eliminating the orchestration overhead of separate draft and target model runs.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Embedding Model Inference
- 
+
 TensorRT-LLM with its optimized XQA attention kernel offers the best performance for embedding models with LLM backbones. FP8 quantization of weights provides performance gains with minimal quality loss, verified by checking cosine similarity between quantized and original output vectors. Embedding models are small enough that horizontal scaling across individual GPU replicas is more effective than multi-GPU parallelism.
@@ -1,5 +1,5 @@
 # Embedding Models
- 
+
 Embedding models convert variable-length text or image input into a fixed-length vector that encodes semantic meaning. These vectors are used in RAG, search, recommendation, and agent memory systems. Embedding inference has two distinct traffic profiles: high-throughput backfills for bulk indexing and low-latency lookups for real-time user queries, which are best served by separate deployments.

 Visit the following resources to learn more:
@@ -9,4 +9,4 @@ Visit the following resources to learn more:
 - [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
 - [@article@Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147)
 - [@article@MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316)
- [@article@What Are Embedding Models? Benefits and Best Practices](https://cohere.com/blog/embedding-models)
+- [@article@What Are Embedding Models? Benefits and Best Practices](https://cohere.com/blog/embedding-models)
@@ -1,3 +1,10 @@
 # Encoder-Decoder Architectures
- 
-Neural networks can be organized as encoder-only, decoder-only, or encoder-decoder models. Modern LLMs are decoder-only. Encoder-only models like BERT are used for text embeddings and classification. Encoder-decoder models like Whisper use an encoder to process input into an internal representation and a decoder to generate output tokens autoregressively. Each architecture has different inference characteristics and tooling requirements.
+
+Neural networks can be organized as encoder-only, decoder-only, or encoder-decoder models. Modern LLMs are decoder-only. Encoder-only models like BERT are used for text embeddings and classification. Encoder-decoder models like Whisper use an encoder to process input into an internal representation and a decoder to generate output tokens autoregressively. Each architecture has different inference characteristics and tooling requirements.
+
+Visit the following resources to learn more:
+
+- [@article@What is an encoder-decoder model?](https://www.ibm.com/think/topics/encoder-decoder-model)
+- [@course@Encoder-Decoder Architecture](https://www.skills.google/course_templates/543)
+- [@video@Encoder-decoder architecture: Overview](https://www.youtube.com/watch?v=zbdong_h-x4)
+- [@video@Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks,](https://www.youtube.com/watch?v=L8HKweZIOmg)
@@ -1,3 +1,7 @@
 # End-to-End Metrics
- 
-End-to-end latency includes inference time, network round-trip, queue time, and client-side overhead. Inference-only metrics tell you how well the model server performs; end-to-end metrics tell you what users actually experience. When inference time is fast but end-to-end time is slow, the bottleneck is in infrastructure or client code, not the GPU.
+
+End-to-end latency includes inference time, network round-trip, queue time, and client-side overhead. Inference-only metrics tell you how well the model server performs; end-to-end metrics tell you what users actually experience. When inference time is fast but end-to-end time is slow, the bottleneck is in infrastructure or client code, not the GPU.
+
+Visit the following resources to learn more:
+
+- [@article@10 LLM Inference Metrics Every AI Engineer Must Know](https://www.intoai.pub/p/llm-inference-metrics)
@@ -1,5 +1,5 @@
 # Expert Parallelism
- 
+
 Expert Parallelism distributes MoE experts across GPUs so each GPU hosts multiple complete experts. Tokens are routed to the GPU holding their activated experts, and each expert runs independently within a single GPU without splitting across multiple. EP has lower inter-GPU communication overhead than Tensor Parallelism and scales well to multi-node deployments via InfiniBand.

 Visit the following resources to learn more:
@@ -1,9 +1,9 @@
 # Fine-Tuning
- 
+
 Fine-tuning takes a pretrained foundation model and adapts it to a specific domain by training on additional labeled data. A small fine-tuned model can match a much larger general model on constrained tasks like text-to-SQL. Smaller models are faster and cheaper to serve, so fine-tuning is one of the highest-leverage tools for hitting latency and cost targets.

 Visit the following resources to learn more:

- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042)
 - [@article@What is Fine-Tuning?](https://www.ibm.com/think/topics/fine-tuning)
 - [@article@How I Fine-Tuned Granite-Vision 2B to Beat a 90B Model](https://towardsdatascience.com/how-i-fine-tuned-granite-vision-2b-to-beat-a-90b-model-insights-and-lessons-learned/?utm_source=roadmap&utm_medium=Referral&utm_campaign=TDS+roadmap+integration)
+- [@video@Is Fine-Tuning Still Needed? LLMs, RAG, & LoRA](https://www.youtube.com/watch?v=-W2JdSl1v48)
@@ -1,5 +1,5 @@
 # FlashAttention
- 
+
 FlashAttention is a series of hand-optimized CUDA kernels that compute attention with fewer reads and writes to GPU memory by fusing the attention algorithm into a single kernel. Standard attention stores intermediate matrices to memory between steps; FlashAttention eliminates those round-trips. FlashAttention 3 targets Hopper GPUs and FlashAttention 4 targets Blackwell GPUs.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Frameworks and Libraries
- 
+
 Frameworks and libraries sit above CUDA and below inference engines, providing the abstractions that inference engineers interact with most directly. This group covers PyTorch and the Hugging Face ecosystem that supply the model implementations and tooling the rest of the stack depends on.
@@ -1,3 +1,3 @@
 # Geo-Aware Load Balancing
- 
+
 A global load balancer routes requests to the GPU cluster geographically closest to the requesting user. Network latency accumulates at roughly 5 milliseconds per time zone traversed, so routing a request from Singapore to San Francisco adds 50 to 100 milliseconds of unnecessary overhead. Geo-aware routing ensures that spare capacity in a distant region is used only when nearby capacity is exhausted.
@@ -1,9 +1,9 @@
 # GPU Architecture
- 
+
 GPU architecture determines the hard limits of what inference can achieve on a given device. This group covers compute, memory, interconnects, and sizing — the four dimensions an inference engineer needs to understand to make informed decisions about hardware selection, parallelism strategy, and optimization priorities.

 Visit the following resources to learn more:

+- [@book@Programming Massively Parallel Processors](https://www.goodreads.com/work/editions/10244675)
 - [@article@NVIDIA Tesla: A Unified Graphics and Computing Architecture](https://ieeexplore.ieee.org/document/4523358)
- [@article@GPU Glossary](https://modal.com/gpu-glossary)
- [@book@Programming Massively Parallel Processors](https://www.goodreads.com/work/editions/10244675)
+- [@article@GPU Glossary](https://modal.com/gpu-glossary)
@@ -1,3 +1,3 @@
 # GPU Models
- 
+
 Knowing the GPU landscape — which devices are available, what they offer, and how they compare — is practical knowledge for procurement decisions, capacity planning, and understanding which optimizations apply to which hardware. This group covers the NVIDIA architectures that dominate inference deployments and the alternatives worth knowing.
@@ -1,3 +1,3 @@
 # GPU Procurement
- 
+
 Cloud GPUs are available from hyperscalers like AWS and GCP, GPU-focused neoclouds like CoreWeave and Nebius, and secondary markets. Procurement mechanisms include reserved instances for predictable baseline load at discounted rates, on-demand instances for flexibility, and spot instances for cost-sensitive batch workloads that tolerate preemption. Large-scale deployments typically combine all three across multiple providers.
@@ -1,5 +1,5 @@
 # Grace and Vera CPUs
- 
+
 NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink Chip-to-Chip at up to 900 GB/s, far higher than standard PCIe connections. This high-bandwidth CPU-GPU link makes offloading KV caches and LoRA weights to CPU memory practical, since data can be retrieved quickly enough to remain on the critical path. Grace pairs with Hopper and Blackwell; Vera pairs with the upcoming Rubin architecture.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # Hopper
- 
+
 The Hopper architecture, released in 2022 with the H100 and later the H200, introduced FP8 Tensor Cores offering twice the compute throughput of FP16 and asynchronous data transfer features used by FlashAttention 3. H100 and H200 GPUs remain among the most widely deployed inference accelerators due to their maturity, broad software support, and relative availability compared to newer generations.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Image and Video Generation Bottlenecks
- 
+
 Image and video generation models are compute-bound because each denoising step applies attention over the entire latent space in parallel, similar to LLM prefill. The models are also relatively small, so memory bandwidth is not the limiting factor. Optimizations focus on FLOPS efficiency rather than memory traffic reduction.
@@ -1,3 +1,3 @@
 # Image and Video Generation
- 
+
 Image and video generation inference is compute-bound rather than memory-bound, making it a different optimization problem from LLM serving. This group covers the production patterns, kernel choices, and parallelism techniques specific to diffusion model deployment.
@@ -1,5 +1,5 @@
 # Image Generation Architecture
- 
+
 Image generation models are pipelines of multiple components: a text encoder that converts the prompt into conditioning, a denoising model that iteratively refines a latent representation from noise into an image, and a variational autoencoder that converts the final latent to pixels. The denoising model is the largest and most compute-intensive component and runs up to 100 forward passes per image generation.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # Image Generation Inference
- 
+
 Production image generation inference uses SGLang Diffusion, TensorRT, or hand-optimized PyTorch. The most important optimization target is the attention kernel, where FlashAttention 3 or 4 outperforms the default on Hopper and Blackwell respectively. GEMM kernels for linear layers can be quantized to FP8 safely. Torch compilation with manual kernel plugins caches the compiled engine to avoid multi-minute recompilation on every cold start.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Independent Component Scaling
- 
+
 Multi-model pipelines, like a voice agent that runs VAD, ASR, an LLM, and TTS in sequence, have components with different hardware requirements and traffic patterns. Each component should autoscale independently based on its own load. Running all components within the same Kubernetes cluster minimizes inter-component network latency, which can otherwise consume a significant fraction of a tight latency budget.
@@ -1,3 +1,3 @@
 # Inference Engines
- 
+
 Inference engines are the core serving layer that takes model weights and handles requests at production scale. They implement continuous batching, memory management, parallelism, and optimization techniques so engineers do not have to implement these from scratch. This group covers the three dominant LLM engines and the orchestration platform that sits above them.
@@ -1,5 +1,5 @@
 # Inference vs. Training
- 
+
 Training is the process of learning model weights from data. Inference is serving those weights in production to generate outputs. Training runs are compute-intensive batch jobs; inference is a continuous, latency-sensitive service. Where training success is measured in loss curves, inference success is measured in time to first token, tokens per second, and uptime.

 Visit the following resources to learn more:
@@ -7,3 +7,4 @@ Visit the following resources to learn more:
 - [@article@Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
 - [@article@Inference vs Training](https://www.cloudflare.com/learning/ai/inference-vs-training/)
 - [@article@What is Machine Learning Inference?](https://hazelcast.com/glossary/machine-learning-inference/)
+- [@video@Why Inference is hard..](https://www.youtube.com/watch?v=B18zBnjZKmc)
@@ -1,5 +1,5 @@
 # Instances and Interconnects
- 
+
 Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, and NVSwitch creates an all-to-all fabric across all eight GPUs. Between nodes, InfiniBand provides inter-node connectivity at substantially lower bandwidth than NVLink. Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated slices, each with dedicated compute and memory, allowing small models to run on fractions of a high-end GPU without resource contention.

 Visit the following resources to learn more:
@@ -1,3 +1,8 @@
 # Inter-Token Latency (ITL)
- 
-ITL is the time between successive generated tokens during decode. An ITL of 10 milliseconds corresponds to 100 tokens per second. It is the per-step measure underlying TPS and is directly tied to how quickly the GPU can complete a single decode forward pass.
+
+ITL is the time between successive generated tokens during decode. An ITL of 10 milliseconds corresponds to 100 tokens per second. It is the per-step measure underlying TPS and is directly tied to how quickly the GPU can complete a single decode forward pass.
+
+Visit the following resources to learn more:
+
+- [@article@Inter-Token Latency - NVIDIA](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html#inter-token-latency)
+- [@article@Inter-Token Latency](https://nhimg.org/glossary/inter-token-latency/)
@@ -1,3 +1,10 @@
 # Inference Engineer
- 
-Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable to serve in production. It spans the full stack from CUDA kernels and GPU architecture to Kubernetes autoscaling and multi-cloud capacity management. The field emerged alongside the explosion of open models and the shift from pay-per-token APIs to dedicated deployments, where engineers own latency, throughput, cost, and uptime directly.
+
+Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable to serve in production. It spans the full stack from CUDA kernels and GPU architecture to Kubernetes autoscaling and multi-cloud capacity management. The field emerged alongside the explosion of open models and the shift from pay-per-token APIs to dedicated deployments, where engineers own latency, throughput, cost, and uptime directly.
+
+Visit the following resources to learn more:
+
+- [@book@Inference Engineering](https://www.baseten.co/inference-engineering/digital-download/)
+- [@article@What Is Inference Engineering? The 2026 GPU Cloud Guide](https://www.spheron.network/blog/inference-engineering-guide-2026/)
+- [@article@A Guide to AI Inference Engineering](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering)
+- [@video@AI Inference: The Secret to AI's Superpowers](https://www.youtube.com/watch?v=XtT5i0ZeHHE&t=19s)
@@ -1,3 +1,3 @@
 # Kernel Selection and Fusion
- 
+
 A CUDA kernel is a function that runs in parallel across thousands of GPU threads simultaneously. Kernel selection is choosing the best existing kernel for a specific operation; kernel fusion combines sequential kernels into one to eliminate unnecessary reads and writes to VRAM between operations. Libraries like cuBLAS, CUTLASS, CuTe, and FlashInfer provide pre-built kernels for common inference operations, and most inference engines perform kernel selection automatically.
@@ -1,5 +1,5 @@
 # KV Cache Storage
- 
+
 KV caches can be stored in GPU VRAM (fastest, tens to hundreds of gigabytes), CPU RAM (slower, hundreds of gigabytes to terabytes), local SSD, or networked SSD. Moving less frequently accessed KV blocks to slower storage frees VRAM for active requests. NVIDIA Dynamo's KV Block Manager provides APIs for managing KV blocks across these storage tiers transparently.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # KV Cache
- 
+
 The KV cache stores the key and value tensors computed during attention for each previously processed token. Without it, attention would require recomputing every prior token on each decode step, making inference prohibitively slow. The KV cache lives in GPU memory and its size grows with sequence length and batch size, making it a major target for memory management techniques.

 Visit the following resources to learn more:
@@ -4,7 +4,7 @@ An LLM is a decoder-only transformer with three main components: an embedding la

 Visit the following resources to learn more:

+- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch)
 - [@article@What is a Large Language Model (LLM)?](https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/)
 - [@article@New to LLMs? Start Here](https://towardsdatascience.com/new-to-llms-start-here/?utm_source=roadmap&utm_medium=Referral&utm_campaign=TDS+roadmap+integration)
- [@video@How Large Language Models Work](https://www.youtube.com/watch?v=5sLYAQS9sWQ)
- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch)
+- [@video@How Large Language Models Work](https://www.youtube.com/watch?v=5sLYAQS9sWQ)
@@ -1,3 +1,9 @@
 # Latency Percentiles
- 
-Mean latency hides outliers. LLM response times follow a right-skewed distribution, so engineers measure P50, P90, P95, and P99. A P99 of several seconds means one in a hundred users has a poor experience even if the median is fast. Reducing P90/P99 latency is as important as reducing average latency.
+
+Mean latency hides outliers. LLM response times follow a right-skewed distribution, so engineers measure P50, P90, P95, and P99. A P99 of several seconds means one in a hundred users has a poor experience even if the median is fast. Reducing P90/P99 latency is as important as reducing average latency.
+
+Visit the following resources to learn more:
+
+- [@article@Percentiles vs averages: why your latency dashboard lies](https://clickhouse.com/resources/engineering/percentiles-vs-averages)
+- [@article@Latency percentiles for load testing analysis](https://gatling.io/blog/latency-percentiles-for-load-testing-analysis)
+- [@video@Mastering Latency Metrics: P90, P95, P99 | System Design](https://www.youtube.com/watch?v=lJ4NEMNBeS4)
@@ -1,7 +1,10 @@
 # Latency vs. Throughput
- 
+
 Lower latency means faster responses per user; higher throughput means more total requests served per unit time. These goals are often in tension. Increasing batch size improves throughput but worsens per-user latency. Choosing the right balance is one of the primary decisions in configuring an inference service.

 Visit the following resources to learn more:

- [@article@The LLM Inference Trilemma: Throughput, Latency, Cost](https://www.digitalocean.com/blog/llm-inference-tradeoffs)
+- [@article@The LLM Inference Trilemma: Throughput, Latency, Cost](https://www.digitalocean.com/blog/llm-inference-tradeoffs)
+- [@article@LLM inference latency: TTFT, tokens per second, and what to measure](https://clickhouse.com/resources/engineering/llm-inference-latency)
+- [@article@⚖️ Latency vs Throughput: Deep Dive into System Performance Trade‑Offs](https://medium.com/@shivanimutke2501/%EF%B8%8F-latency-vs-throughput-deep-dive-into-system-performance-trade-offs-adcd5860dc56)
+- [@video@Latency vs Throughput in 60 Seconds](https://www.youtube.com/watch?v=azUdQ9bme9g)
@@ -1,8 +1,8 @@
 # LLM Inference Bottlenecks
- 
+
 LLM prefill is compute-bound because it processes the entire input sequence as one large matrix multiplication, performing many operations per byte loaded. LLM decode is memory-bound because it generates tokens one at a time via vector-matrix multiplications that require loading all model weights from memory for each token while performing relatively few arithmetic operations.

 Visit the following resources to learn more:

- [@article@GPU Glossary](https://modal.com/gpu-glossary)
- [@book@AI Systems Performance Engineering](https://www.oreilly.com/library/view/ai-systems-performance/9781098170202/)
+- [@book@AI Systems Performance Engineering](https://www.oreilly.com/library/view/ai-systems-performance/9781098170202/)
+- [@article@GPU Glossary](https://modal.com/gpu-glossary)
@@ -1,3 +1,3 @@
 # Local and Edge Inference
- 
+
 Not all inference runs in a datacenter. Desktop and mobile deployments have different hardware constraints, tooling ecosystems, and use cases from cloud inference. This group covers the patterns and tools for running models on consumer hardware and mobile devices.
@@ -1,5 +1,5 @@
 # Long Context Handling
- 
+
 Long sequences produce KV caches large enough to strain GPU memory, slow down decode, and increase prefill time. Techniques for handling long context include FlashAttention for memory-efficient attention computation, PagedAttention for non-contiguous KV storage, chunked prefill to interleave long prefill with decode, and model-level approaches like sliding window attention that limit how many prior tokens each token attends to.

 Visit the following resources to learn more:
@@ -8,4 +8,4 @@ Visit the following resources to learn more:
 - [@article@RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864)
 - [@article@What is Long Context and Why Does it Matter for AI?](https://cloud.google.com/transform/the-prompt-what-are-long-context-windows-and-why-do-they-matter)
 - [@article@Long-Context LLM Infrastructure: Building Systems for Million-Token Windows](https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide)
- [@video@Deep Dive into Long Context](https://www.youtube.com/watch?v=NHMJ9mqKeMQ)
+- [@video@Deep Dive into Long Context](https://www.youtube.com/watch?v=NHMJ9mqKeMQ)
@@ -1,3 +1,3 @@
 # LoRA-Aware Routing
- 
+
 LoRA fine-tune weights are small adapter layers loaded on top of a base model to specialize its behavior for a specific task or customer. A deployment serving many LoRA variants for the same base model must route each request to a replica that already has the required LoRA loaded in GPU memory. Routing without this awareness forces constant LoRA swapping, adding latency and reducing throughput on each affected replica.
@@ -1,3 +1,3 @@
 # Measuring Quality Impact
- 
+
 After quantizing a model, quality must be validated against the original. Three methods are used: perplexity scoring (lower is better, with minimal increase after quantization), standard intelligence benchmarks like MMLU or SWE-bench, and product-specific evals tailored to the actual use case. All three should show differences indistinguishable from noise before a quantized model is deployed.
@@ -1,3 +1,3 @@
 # Memory
- 
+
 GPU memory determines what fits on a single device and how fast data can move through the compute units. VRAM capacity sets the upper limit on model size and KV cache; VRAM bandwidth determines how quickly the decode phase can load weights per token. Modern datacenter GPUs use high-bandwidth memory (HBM) in generations HBM3, HBM3e, and HBM4, stacked directly on the GPU die to maximize bandwidth.
@@ -1,5 +1,5 @@
 # Mixture of Experts (MoE)
- 
+
 Mixture of Experts models replace dense linear layers with a set of smaller expert matrices, activating only a subset of experts per token via a learned router. A model like Qwen3-235B activates only 22 billion of its 235 billion parameters per forward pass, making single-user inference efficient. In batched production serving, different tokens activate different experts, so nearly all parameters are active across the batch.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Mobile Inference
- 
+
 Mobile devices run inference using vendor frameworks: Google's AI Edge SDK for Android and Apple's Foundation Models and Core ML for iOS. Hardware limitations restrict practical model sizes to one to two billion parameters on current phones. Small fine-tuned models for tasks like transcription, translation, and on-device classification are the strongest fit for mobile edge inference.
@@ -1,11 +1,10 @@
 # Model Evaluation
- 
+
 Model evaluation, or evals, is the practice of systematically measuring model quality for a specific task or domain. Standard intelligence benchmarks like MMLU measure general capabilities but can be gamed; product-specific evals that test real use cases provide more reliable signal. High-confidence evals are a prerequisite before investing in inference optimization, ensuring the model is actually useful before making it fast.

 Visit the following resources to learn more:

- [@article@How to Fine-Tune Qwen3 to GPT-4o Level Performance](https://ghost.oxen.ai/how-to-fine-tune-qwen3-to-gpt-4o-level-performance/)
 - [@article@LLM Evaluation: A Beginner's Guide](https://www.evidentlyai.com/llm-guide/llm-evaluation)
 - [@article@LLM Evaluation Metrics and Methods, Explained Simply](https://www.evidentlyai.com/llm-guide/llm-evaluation-metrics)
- [@article@A Pragmatic Guide to LLM Evals for Devs](https://newsletter.pragmaticengineer.com/p/evals)
- [@book@Evals for AI Engineers](https://www.oreilly.com/library/view/evals-for-ai/9798341660717/)
+- [@article@Automated Regression Testing: The Complete 2026 Guide](https://roadmap.sh/ai-engineer/automated-regression-testing)
+- [@video@LLM as a Judge: Scaling AI Evaluation Strategies](https://www.youtube.com/watch?v=trfUBIDeI1Y)
@@ -1,3 +1,5 @@
+# undefined
+
 ### Model Formats and Runtimes
- 
+
 Before a model can serve requests, its weights must be stored, loaded, and compiled into an executable form. This group covers the file formats used to distribute weights and the runtimes that compile them into optimized inference engines for specific hardware targets.
@@ -1,3 +1,3 @@
 # Model Parallelism
- 
+
 When a model is too large to fit on a single GPU, or when serving latency requires more compute than one device can provide, model parallelism distributes the workload across multiple GPUs or nodes. This group covers the three parallelism strategies and their trade-offs in terms of communication overhead, latency, and hardware requirements.
@@ -1,9 +1,10 @@
 # Model Selection
- 
+
 The most important decision in model performance optimization is which model to use. Smaller models with fewer parameters are faster and cheaper to serve than larger ones. The goal is to find the smallest model that meets quality requirements for the task. Model architecture also determines which inference engines and optimization techniques are available, so popular architectures with broad tooling support are preferable.

 Visit the following resources to learn more:

 - [@article@Choosing the Right Model](https://bentoml.com/llm/getting-started/choosing-the-right-model)
 - [@article@Beyond Vibes: How to Properly Select the Right LLM for the Right Task](https://aws.amazon.com/blogs/machine-learning/beyond-vibes-how-to-properly-select-the-right-llm-for-the-right-task/)
- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/)
+- [@video@How to Pick the Right AI Foundation Model](https://www.youtube.com/watch?v=pePAAGfh-IU)
+- [@video@How to Choose Large Language Models: A Developer’s Guide to LLMs](https://www.youtube.com/watch?v=pYax2rupKEY)
@@ -1,3 +1,3 @@
 # ModelOpt
- 
+
 ModelOpt is NVIDIA's toolkit for post-training quantization, supporting FP8, INT8, FP4, MXFP8, MXFP4, and NVFP4 output formats. It integrates with PyTorch and exports quantized models directly to TensorRT-LLM, vLLM, and SGLang. Calibration runs a small set of representative inputs through the model to compute per-layer scale factors that minimize the gap between quantized and original outputs.
@@ -1,10 +1,10 @@
 # Multi-Cloud
- 
+
 Multi-cloud inference distributes GPU capacity across providers and regions to maximize availability, minimize latency, and meet data residency requirements. This group covers procurement strategy, capacity management, geographic routing, and the reliability patterns that make large-scale inference resilient to hardware and provider failures.

 Visit the following resources to learn more:

- [@article@SemiAnalysis](https://semianalysis.com/)
- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
 - [@book@Designing Data-Intensive Applications](https://dataintensive.net/)
- [@book@Site Reliability Engineering](https://sre.google/books/)
+- [@book@Site Reliability Engineering](https://sre.google/books/)
+- [@article@SemiAnalysis](https://semianalysis.com/)
+- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
@@ -1,3 +1,3 @@
 # Multi-Node Inference
- 
+
 Multi-node inference spreads a model across more than eight GPUs using InfiniBand for inter-node communication. It is necessary for the largest models where a single node lacks sufficient VRAM for weights and KV cache. InfiniBand bandwidth is an order of magnitude lower than NVLink, so parallelism strategies must minimize cross-node communication. For MoE models, Expert Parallelism is preferred over Tensor Parallelism between nodes.
@@ -1,5 +1,5 @@
 # N-gram Speculation
- 
+
 N-gram speculation builds a dictionary during prefill that maps token prefixes to observed n-gram continuations from the input. During decode, generated tokens are looked up in this dictionary and matching suffixes become draft tokens with no separate model required. It is most effective for code completion, where output frequently mirrors input syntax, and can produce very long draft sequences with high acceptance rates in that domain.

 Visit the following resources to learn more:
@@ -1,9 +1,10 @@
 # Neural Networks
- 
+
 A neural network is a layered structure of nodes where each node applies a learned weight matrix and bias to its input. Multiple layers connected in sequence form the foundation for all generative AI models. Inference engineers need a working model of how neural networks compute outputs to understand where performance bottlenecks arise.

 Visit the following resources to learn more:

 - [@book@Deep Learning](https://www.deeplearningbook.org/)
 - [@book@Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python-second-edition)
- [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch)
+- [@article@What is deep learning?](https://www.ibm.com/think/topics/deep-learning)
+- [@video@But what is a neural network?](https://www.youtube.com/watch?v=aircAruvnKk)
@@ -1,3 +1,3 @@
 # NIMs
- 
+
 NIMs are pre-built Docker containers provided by NVIDIA for popular open models, available as flexible multi-LLM containers or as GPU- and model-specific optimized containers. They provide a ready-made starting point for serving common architectures without building a custom container from scratch. For maximum control and custom configurations, starting from a less opinionated base image and building up is generally preferable.
@@ -1,5 +1,5 @@
 # NVIDIA Dynamo
- 
+
 Dynamo supports conditional disaggregation, where requests are first sent to the decode engine and only routed to a prefill engine when the input is long or not already cached. The ratio of prefill to decode engines is configurable at runtime and can shift as traffic patterns change. This dynamic allocation prevents the prefill queue from becoming a bottleneck and avoids wasting decode engine resources on prefill-heavy traffic.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # NVIDIA Dynamo
- 
+
 Dynamo is NVIDIA's open-source distributed serving platform that sits above inference engines and provides orchestration for large-scale deployments. It adds KV-aware routing, disaggregated prefill/decode serving, multi-node parallelism, and an SLA-based planner that dynamically adjusts the number of prefill and decode workers based on TTFT and TPS targets.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # NVIDIA
- 
+
 NVIDIA GPUs are the dominant hardware for inference due to their combination of Tensor Core compute, HBM bandwidth, NVLink interconnects, and the CUDA software ecosystem. The three most relevant architectures for current inference deployments are Hopper, Blackwell, and the Grace and Vera CPU platforms that pair with them.
@@ -1,5 +1,5 @@
 # Observability
- 
+
 Inference observability tracks total request volume, input and output sequence lengths, response codes, latency percentiles (P50, P90, P99 of TTFT, TPS, and end-to-end latency), replica count, GPU utilization, and queue depth. These metrics are interdependent; a latency spike could come from long input sequences, high concurrency, or a hardware issue. Observability must integrate with existing tooling like Grafana, Datadog, and PagerDuty to put inference metrics in context.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Omni-Modal Models
- 
+
 Omni models accept multiple input modalities and produce multiple output types in a single model. They avoid the latency of running separate specialized models in sequence but often lag behind dedicated models in quality within each modality. Production pipelines using VLMs typically coordinate preprocessing steps like OCR, audio transcription, and PDF extraction as separate independently scaling components.
@@ -1,3 +1,9 @@
 # Online vs. Offline Inference
- 
-Online inference serves real-time user-facing requests where latency matters. Offline inference processes large batches of data asynchronously where throughput and cost matter more than per-request speed. The same model, like Whisper for transcription, may need separate deployments for each use case because the optimal configurations differ significantly.
+
+Online inference serves real-time user-facing requests where latency matters. Offline inference processes large batches of data asynchronously where throughput and cost matter more than per-request speed. The same model, like Whisper for transcription, may need separate deployments for each use case because the optimal configurations differ significantly.
+
+Visit the following resources to learn more:
+
+- [@article@Production ML systems: Static versus dynamic inference](https://developers.google.com/machine-learning/crash-course/production-ml-systems/static-vs-dynamic-inference)
+- [@article@Trade-offs in Online vs Offline Model Inference](https://bugfree.ai/knowledge-hub/trade-offs-online-vs-offline-model-inference)
+- [@video@Offline Recurrence for Improved Online Inference](https://www.youtube.com/watch?v=bwvE1KvLzLQ)
@@ -1,5 +1,5 @@
 # ONNX Runtime
- 
+
 ONNX Runtime is an open-source inference runtime that executes models exported to the ONNX format, which stores both weights and a computation graph for cross-platform portability. It supports multiple hardware backends including CUDA, DirectML, and CPU. It is widely used for embedding models, vision models, and pipelines that need to run across hardware environments without engine-specific configuration.

 Visit the following resources to learn more:
@@ -1,9 +1,10 @@
 # Open vs. Closed Models
- 
+
 Closed models are accessed exclusively through a vendor API; the weights are never released. Open models publish their weights for download and self-hosting. For inference engineers, this distinction determines the entire problem space: closed models require no infrastructure but offer no control over latency, cost, or configuration, while open models require building and operating a full inference stack but give complete control over every layer of it.

 Visit the following resources to learn more:

+- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/)
+- [@article@Open-source Models vs. Closed-Source Models: A Simple Guide](https://koat.ai/blog/open-source-models-vs-closed-source-models-a-simple-guide)
 - [@article@Open-Source vs. Closed LLMs: A Guide](https://hatchworks.com/blog/gen-ai/open-source-vs-closed-llms-guide/)
- [@video@Open Source vs Closed AI: LLMs, Agents & the AI Stack Explained](https://www.youtube.com/watch?v=_QfxGZGITGw)
- [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/)
+- [@video@Open Source vs Closed AI: LLMs, Agents & the AI Stack Explained](https://www.youtube.com/watch?v=_QfxGZGITGw)
@@ -1,3 +1,3 @@
 # Other Accelerators
- 
+
 Alternatives to NVIDIA GPUs include AMD MI350, Google TPUs, AWS Inferentia and Trainium, Cerebras WSE-3, and Groq LPUs. Each offers a specific advantage: AMD provides CUDA-compatible tooling via ROCm; TPUs integrate deeply with Google Cloud and JAX; Inferentia and Trainium offer cost advantages within AWS. All face the shared challenge of building an ecosystem to match the breadth of NVIDIA's CUDA libraries and inference engine support.
@@ -1,5 +1,5 @@
 # PagedAttention
- 
+
 PagedAttention stores the KV cache in fixed-size pages rather than a single contiguous memory block, allowing it to be distributed across GPU memory without fragmentation. It uses a lookup table to access pages on demand, reducing memory waste from over-allocation and enabling more requests to share GPU memory. It is the memory management technique that made vLLM's high throughput possible.

 Visit the following resources to learn more:
@@ -1,3 +1,3 @@
 # Performance Benchmarking
- 
+
 Benchmarking measures inference service performance under realistic workloads. Good benchmarks simulate realistic input and output sequence lengths, concurrency patterns, and request contents rather than synthetic uniform inputs. Standard tools include SGLang Genai-bench, NVIDIA GenAI-Perf, and Locust. Always establish a baseline before applying optimizations and change one variable at a time.
@@ -1,5 +1,5 @@
 # Pipeline Parallelism
- 
+
 Pipeline Parallelism splits an LLM's layers into sequential stages and assigns each stage to a different GPU or node. Unlike Tensor Parallelism, each GPU holds complete layers rather than layer fragments. It introduces pipeline bubbles where GPUs wait for preceding stages, reducing efficiency. It is used for multi-node inference with dense models, typically combined with Tensor Parallelism within each node.

 Visit the following resources to learn more:
@@ -1,3 +1,9 @@
 # Prefill and Decode Phases
- 
-LLM inference has two distinct phases. Prefill processes the entire input sequence in parallel to build the KV cache and is compute-bound. Decode generates output tokens one at a time by reading model weights from memory and is memory-bound. Each phase has different bottlenecks and responds to different optimization techniques.
+
+LLM inference has two distinct phases. Prefill processes the entire input sequence in parallel to build the KV cache and is compute-bound. Decode generates output tokens one at a time by reading model weights from memory and is memory-bound. Each phase has different bottlenecks and responds to different optimization techniques.
+
+Visit the following resources to learn more:
+
+- [@article@Prefill vs Decode: LLM Inference Phases Explained](https://redis.io/blog/prefill-vs-decode/)
+- [@article@Understanding the Two Key Stages of LLM Inference: Prefill and Decode(Part-1)](https://medium.com/@sailakkshmiallada/understanding-the-two-key-stages-of-llm-inference-prefill-and-decode-29ec2b468114)
+- [@video@Why LLMs Read Fast but Write Slowly - Prefill vs Decode](https://www.youtube.com/watch?v=hmMgTgQpO38)
@@ -1,5 +1,5 @@
 # Profiling Performance
- 
+
 Profiling reveals where time is spent within a benchmark, identifying which operations consume the most compute or memory. Tools include PyTorch Profiler, NVIDIA Nsight Systems for system-wide GPU and CPU analysis, and NVIDIA Nsight Compute for per-kernel analysis. Profiling is most useful when writing custom inference code or contributing to inference frameworks, rather than configuring existing engines.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # PyTorch
- 
+
 PyTorch is the standard Python framework for building and running neural networks. It abstracts CUDA into a tensor operation API and provides automatic differentiation for training. For inference, PyTorch supports torch.compile for automatic kernel optimization and serves as the foundation that all major inference engines build on top of.

 Visit the following resources to learn more:
@@ -1,5 +1,5 @@
 # Quantization
- 
+
 Quantization converts model weights and other values from their native precision to a lower-precision format. Cutting precision in half doubles memory bandwidth for decode and enables twice the FLOPS for prefill on compatible Tensor Cores. The tradeoff is potential quality loss from reduced numerical precision, which can compound across the many calculations in a forward pass.

 Visit the following resources to learn more:
@@ -1,3 +1,9 @@
 # Real-Time Factor (RTF)
- 
-RTF measures how quickly an ASR system can process audio relative to its duration. A system that transcribes one hour of audio in six seconds has an RTF of 600x. It is the standard throughput metric for long-file transcription workloads and improves roughly linearly with the number of GPUs used for parallel chunk processing.
+
+RTF measures how quickly an ASR system can process audio relative to its duration. A system that transcribes one hour of audio in six seconds has an RTF of 600x. It is the standard throughput metric for long-file transcription workloads and improves roughly linearly with the number of GPUs used for parallel chunk processing.
+
+Visit the following resources to learn more:
+
+- [@article@What is RTF (Real-Time Factor) in ASR?    Copied from: https://www.futurebeeai.com/knowledge-hub/real-time-factor-asr](https://www.futurebeeai.com/knowledge-hub/real-time-factor-asr)
+- [@article@Whisper Real-Time Factor by GPU](https://gigagpu.com/whisper-rtf-by-gpu/)
+- [@article@RTF — Real-Time Factor Explained](https://transcript.you/glossary/rtf)
@@ -1,7 +1,10 @@
 # Reasoning Models
- 
+
 Reasoning models generate an intermediate thinking sequence before producing their final output, adding a third token sequence alongside the input and output. This reasoning trace increases total token count significantly and changes inference economics: TTFT is higher, total latency is longer, and output token costs dominate. Inference engineers must account for the variable and sometimes very long reasoning sequences when sizing infrastructure and setting latency budgets.

 Visit the following resources to learn more:

- [@article@Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)
+- [@article@Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)
+- [@article@What is a reasoning model?](https://www.ibm.com/think/topics/reasoning-model)
+- [@video@What Are Large Reasoning Models (LRMs)? Smarter AI Beyond LLMs](https://www.youtube.com/watch?v=enLbj0igyx4)
+- [@video@How do thinking and reasoning models work?](https://www.youtube.com/watch?v=xCRvOUykOX0)
@@ -1,3 +1,3 @@
 # Request Queueing
- 
+
 A request queue holds incoming traffic when all active replicas are at capacity while autoscaling brings new replicas online. A first-in, first-out queue is the standard implementation; priority queues can give paid users better position. Queues must be visible to newly started replicas so that requests are not held waiting for existing replicas when new capacity becomes available.
@@ -1,9 +1,9 @@
 # Routing and Load Balancing
- 
+
 Routers make request-level decisions about which replica to send a request to, considering factors like KV cache prefix match and LoRA availability. Load balancers make system-level decisions to distribute traffic evenly across replicas. In systems with prefix caching, naive round-robin load balancing misses cache opportunities; intelligent routing increases cache hit rates and reduces TTFT.

 Visit the following resources to learn more:

- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
 - [@book@Designing Data-Intensive Applications](https://dataintensive.net/)
- [@book@Site Reliability Engineering](https://sre.google/books/)
+- [@book@Site Reliability Engineering](https://sre.google/books/)
+- [@article@The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
@@ -1,3 +1,3 @@
 # Safetensors
- 
+
 Safetensors is the standard format for storing and distributing open model weights. It uses memory mapping for fast, zero-copy loading and prevents execution of arbitrary code during deserialization, making it safer than earlier pickle-based formats. All major inference engines load safetensors natively, and most open models on Hugging Face publish weights in this format.
@@ -1,3 +1,3 @@
 # Scale to Zero
- 
+
 Scale to zero allows the system to run zero replicas during periods of no traffic and spin up on demand. It requires fast cold starts and robust queuing to hold incoming requests while a replica starts. Scale to zero is well-suited for development environments and periodically accessed workloads, but not for latency-sensitive production applications with unpredictable light traffic.
@@ -1,3 +1,3 @@
 # Security and Compliance
- 
+
 Inference infrastructure must protect user inputs, model outputs, and model weights. Standard practices include data encryption, container security, network isolation, and access controls validated by third-party penetration testing. Regulated industries require compliance with frameworks like SOC 2 and HIPAA, which also apply to underlying cloud providers. Multi-cloud deployments support data residency requirements by routing each country's data to clusters in that country.
--- a/Show More
+++ b/Show More