chore: sync content to repo (#10398)

Co-authored-by: nilbuild <4921183+nilbuild@users.noreply.github.com>
2026-10-24 01:08:07 +08:00 · 2026-09-30 17:28:38 +02:00
parent a5a37a3812
commit d7e1090611
140 changed files with 420 additions and 0 deletions
@@ -0,0 +1,3 @@
+# AI-Native Applications
+ 
+AI-native applications are products where generative models power the core user experience, from code editors and voice agents to customer support and media generation. Each category has different inference requirements. Agents generate many inference calls per user action; chat applications prioritize TTFT; voice applications need low end-to-end latency for natural conversation.
@@ -0,0 +1,3 @@
+# Arithmetic Intensity
+
+Arithmetic intensity is a property of an algorithm: the number of floating-point operations performed per byte of memory read or written. A large matrix multiplication has high arithmetic intensity because the same weights participate in many operations per load; a decode forward pass has low arithmetic intensity because each weight is loaded from VRAM and used once per token. Comparing a workload's arithmetic intensity against the GPU's ops:byte ratio is the fastest way to diagnose whether adding compute or adding bandwidth would help.
@@ -0,0 +1,3 @@
+# Asynchronous Inference
+ 
+Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing.
@@ -0,0 +1,3 @@
+# Asynchronous Inference
+ 
+Asynchronous inference is a fire-and-forget request model where the client submits a job, receives an immediate acknowledgment, and retrieves the result later via a webhook. It is suited for throughput-sensitive, latency-insensitive workloads like bulk document embedding or corpus transcription. Async jobs have much longer time limits than synchronous requests and require robust server-side queuing.
@@ -0,0 +1,3 @@
+# Attention Mechanism
+ 
+Attention is the mechanism transformers use to relate each token to all previous tokens via query, key, and value projections. During decode, each new token attends to every prior token, and the KV cache stores precomputed keys and values to avoid recomputation. Attention is the most expensive and most heavily optimized operation in LLM inference.
@@ -0,0 +1,3 @@
+# Attention Optimization
+
+Attention optimization refers to techniques designed to improve the computational efficiency and memory usage of the attention mechanism within transformer models. Because standard attention scales quadratically with input sequence length, these optimizations aim to reduce computation and memory overhead. By streamlining how tokens interact with one another, these methods allow models to process longer inputs faster and run on hardware with less available memory.
@@ -0,0 +1,3 @@
+# Attention Variants
+ 
+Newer attention algorithms reduce the quadratic time complexity of standard attention. Sliding window attention limits each token to attending to the nearest w tokens, turning O(N²) into O(Nw). Linear attention approximates the softmax with a linear-time kernel. Compressed attention periodically compresses earlier context. These variants trade off some quality for better scaling on long sequences.
@@ -0,0 +1,3 @@
+# Automatic Speech Recognition (ASR)
+ 
+ASR models like Whisper take audio input and produce text transcripts. Whisper is an encoder-decoder model where the decoder, an autoregressive transformer, dominates inference time. TensorRT-LLM with in-flight batching for the decoder delivers the best performance. For real-time transcription, streaming via WebSockets with VAD-segmented chunks achieves round-trip targets of around 200 milliseconds per chunk on optimized deployments.
@@ -0,0 +1,3 @@
+# Autoscaling
+ 
+Autoscaling dynamically adjusts the number of active model replicas to match incoming traffic, maintaining latency SLAs while minimizing idle GPU spend. Kubernetes is the standard orchestration layer, scaling replicas within a cluster based on traffic signals and utilization metrics. Key configuration parameters are min replicas, max replicas, the autoscaling window, the scale-down delay, and the concurrency target per replica.
@@ -0,0 +1,3 @@
+# Benchmarking and Profiling
+ 
+Benchmarking and profiling are how inference engineers turn intuition into evidence. Benchmarking measures system-level throughput and latency under load; profiling zooms in to identify which specific operations are consuming time. Both are prerequisites for making optimization decisions with confidence.
@@ -0,0 +1,3 @@
+# BERT-style Models
+
+BERT-style encoder-only models are the traditional architecture for text embeddings, producing a fixed-size vector by pooling the final hidden states of a bidirectional transformer. They are compact, fast, and well-suited for high-throughput indexing workloads where embedding quality is sufficient for the task. Models like BGE-small and all-MiniLM run efficiently on CPU or fractional GPU instances, making them the lowest-cost embedding option.
@@ -0,0 +1,3 @@
+# Blackwell
+ 
+The Blackwell architecture, released in late 2024 with the B200 and B300, added FP4 Tensor Cores and new microscaling number formats including MXFP8, MXFP4, and NVFP4. B200 GPUs offer approximately 5 petaFLOPS of FP8 compute and up to 8 TB/s of memory bandwidth. They are the current gold standard for demanding inference workloads including large LLMs and video generation.
@@ -0,0 +1,3 @@
+# Bottleneck Analysis
+
+Bottleneck analysis identifies whether a given inference workload is limited by compute or by memory bandwidth, which determines which optimizations will actually improve performance. The core tool is the ops:byte ratio: comparing a GPU's peak FLOPS against its peak memory bandwidth reveals the hardware's balance point, and comparing a model operation's arithmetic intensity against that balance point reveals whether it is compute-bound or memory-bound. LLM prefill and image generation are compute-bound; LLM decode is memory-bound.
@@ -0,0 +1,3 @@
+# Cache-Aware Routing
+ 
+In deployments with multiple replicas, cache-aware routing directs requests to the replica most likely to have a matching prefix already in its KV cache. Without this, a user in a long conversation may hit different replicas each turn, getting cache misses that increase latency. A global KV cache stored on networked SSD allows all replicas to eventually access any cached sequence.
@@ -0,0 +1,3 @@
+# Caching Approaches
+ 
+KV cache reuse is the primary caching strategy in LLM inference, covering both prefix caching (reusing cached KV values across requests that share a common prefix) and KV cache offloading (moving less-used cache blocks to slower storage tiers to free GPU VRAM). A related concern is cache-aware routing, which directs requests to replicas most likely to already hold the relevant cached prefix.
@@ -0,0 +1,3 @@
+# Caching
+ 
+Caching reuses previously computed KV values to avoid redundant prefill computation across requests that share common context. It is one of the most impactful techniques for reducing TTFT in workloads with long system prompts or multi-turn conversations. This group covers the approaches, storage tiers, routing implications, and long-context strategies that make caching effective at scale.
@@ -0,0 +1,3 @@
+# Capacity Management
+ 
+Multi-cloud inference pools GPU capacity from multiple cloud providers and regions into a single schedulable resource, treating distinct compute pools as fungible. This requires a global control plane for deployment and scaling decisions alongside per-cluster workload planes that serve traffic and report utilization. True multi-cloud unlocks greater total capacity, improved reliability through redundancy, lower latency via geographic proximity, and data sovereignty compliance.
@@ -0,0 +1,3 @@
+# Client Integration
+ 
+The client layer connects application code to the inference service and contributes its own slice of end-to-end latency. This group covers the connection overhead, async request patterns, and protocol choices that determine how efficiently client and server communicate.
@@ -0,0 +1,3 @@
+# Client Latency Overhead
+ 
+Establishing a new connection between a client and inference server takes tens of milliseconds for TLS handshake, which can represent ten percent or more of a tight latency budget. Session reuse across requests eliminates this overhead for returning clients. Standard SDKs like the OpenAI SDK handle session reuse automatically; custom clients must implement it explicitly.
@@ -0,0 +1,3 @@
+# Cold Starts
+ 
+A cold start is the time from zero replicas to serving the first request, including GPU provisioning, container image loading, model weight loading, and inference engine startup or compilation. Reducing cold start time is essential for autoscaling systems to scale down confidently and scale up responsively. Key techniques include quantized weights, local weight caching near the GPU instances, and caching compiled inference engines to disk.
@@ -0,0 +1,3 @@
+# Compute
+ 
+A GPU's compute capability is determined by its Streaming Multiprocessors (SMs), each containing CUDA Cores for general arithmetic and Tensor Cores for matrix multiply-accumulate operations. Tensor Cores are the primary resource for inference, executing the matmuls that make up the majority of forward pass computation. GPU compute is measured in peak Tensor Core FLOPS at a given precision, and roughly doubles with each new NVIDIA architecture generation.
@@ -0,0 +1,3 @@
+# Consumer vs. B2B Workloads
+ 
+Consumer applications are cost-sensitive and have unpredictable traffic spikes, requiring flexible autoscaling and low marginal cost. Business-to-business applications prioritize uptime and consistent low latency, as they often sit in the critical path of revenue-generating workflows. Compliance requirements, including data residency and regulatory certifications, are more common in enterprise deployments.
@@ -0,0 +1,3 @@
+# Containerization
+ 
+Containerization packages an inference service and all its dependencies into a Docker image that runs consistently across environments. Inference containers include the CUDA toolkit, Python packages, the inference engine, and any system libraries like ffmpeg for audio processing. Using a pinned dependency tree prevents breaking changes from affecting running services and ensures reproducible builds.
@@ -0,0 +1,3 @@
+# Continuous Batching
+ 
+Continuous batching operates at the token level, swapping requests in and out of the batch as they complete individual tokens rather than waiting for a full batch to finish. This keeps GPU utilization high and minimizes the latency penalty of batching compared to static or dynamic batching. All major inference engines implement continuous batching by default.
@@ -0,0 +1,3 @@
+# Cost Estimation
+ 
+Dedicated inference cost depends on batch sizing, traffic patterns, and sequence lengths rather than simply on token count. Comparing dedicated versus API costs requires converting per-token API pricing into total cost over a representative time window and comparing it to GPU-hour costs at the same traffic volume. Total cost of ownership also includes engineering time for building and maintaining the inference system.
@@ -0,0 +1,3 @@
+# CUDA
+ 
+CUDA is NVIDIA's proprietary platform for programming GPUs, consisting of a kernel language (C++), a driver for hardware management, and a runtime API for launching kernels. Every inference engine, framework, and library in the NVIDIA ecosystem depends on CUDA. Understanding CUDA is the foundation for diagnosing performance problems that tools and engines cannot solve automatically.
@@ -0,0 +1,3 @@
+# Dependency Management
+ 
+Inference dependency chains are long and fragile due to the tight coupling between CUDA versions, framework versions, and hardware architectures. Pinning exact versions of every dependency prevents unexpected breakage when upstream packages release updates. Containers for newly released models often initially depend on pre-release software builds and need to be rebuilt on stable releases within days of the model launch.
@@ -0,0 +1,3 @@
+# Desktop Inference
+ 
+Desktop inference runs on consumer hardware, primarily Apple M-series unified memory systems and high-end NVIDIA consumer GPUs. Apple hardware offers very large unified memory capacity at moderate bandwidth, enabling larger models than discrete consumer GPUs; NVIDIA consumer GPUs offer higher bandwidth with smaller capacity. Tools like Ollama and llama.cpp make it practical to run aggressively quantized models on consumer hardware with minimal setup.
@@ -0,0 +1,3 @@
+# Diffusion Models
+ 
+Diffusion models generate images by starting from random noise and iteratively denoising it over 30 to 50 steps, guided by a text prompt. Each step runs the denoiser twice, once with conditioning and once without, to apply classifier-free guidance. The entire process operates in a low-dimensional latent space rather than full pixel space to make attention computationally feasible.
@@ -0,0 +1,3 @@
+# Disaggregation
+ 
+Disaggregation separates the prefill and decode phases onto independent GPU instances that can scale separately. The prefill engine processes the input sequence and sends the resulting KV cache to the decode engine via the hardware interconnect. This removes the resource contention between compute-heavy prefill and memory-heavy decode. Disaggregation is most effective at high traffic volumes, for large models, and with prefill-heavy workloads like code editors.
@@ -0,0 +1,3 @@
+# Distillation
+ 
+Distillation trains a smaller student model to emulate a larger teacher model by exposing it to the teacher's probability distributions, not just its final outputs. Unlike fine-tuning on synthetic data, distillation transfers the teacher's reasoning behavior. The resulting student model is smaller and faster while retaining much of the teacher's quality on the same tasks.
@@ -0,0 +1,3 @@
+# Draft-Target Speculative Decoding
+ 
+Draft-target speculation uses a small draft model, typically at least ten times smaller than the target, to generate draft tokens for the target model to verify. The draft model should share the same tokenizer and general behavior as the target, often being a smaller member of the same model family. It is the simplest speculation method to set up but introduces the most overhead due to running a separate model.
@@ -0,0 +1,3 @@
+# EAGLE
+ 
+EAGLE is a purpose-built speculation method where a small draft model is trained to accept hidden states from the target model as input. By seeing the target's internal representations, EAGLE achieves higher acceptance rates and longer draft sequences (up to eight tokens) than using a general-purpose small model. It integrates into the same forward pass as the target model, eliminating the orchestration overhead of separate draft and target model runs.
@@ -0,0 +1,3 @@
+# Embedding Model Inference
+ 
+TensorRT-LLM with its optimized XQA attention kernel offers the best performance for embedding models with LLM backbones. FP8 quantization of weights provides performance gains with minimal quality loss, verified by checking cosine similarity between quantized and original output vectors. Embedding models are small enough that horizontal scaling across individual GPU replicas is more effective than multi-GPU parallelism.
@@ -0,0 +1,3 @@
+# Embedding Models
+ 
+Embedding models convert variable-length text or image input into a fixed-length vector that encodes semantic meaning. These vectors are used in RAG, search, recommendation, and agent memory systems. Embedding inference has two distinct traffic profiles: high-throughput backfills for bulk indexing and low-latency lookups for real-time user queries, which are best served by separate deployments.
@@ -0,0 +1,3 @@
+# Encoder-Decoder Architectures
+ 
+Neural networks can be organized as encoder-only, decoder-only, or encoder-decoder models. Modern LLMs are decoder-only. Encoder-only models like BERT are used for text embeddings and classification. Encoder-decoder models like Whisper use an encoder to process input into an internal representation and a decoder to generate output tokens autoregressively. Each architecture has different inference characteristics and tooling requirements.
@@ -0,0 +1,3 @@
+# End-to-End Metrics
+ 
+End-to-end latency includes inference time, network round-trip, queue time, and client-side overhead. Inference-only metrics tell you how well the model server performs; end-to-end metrics tell you what users actually experience. When inference time is fast but end-to-end time is slow, the bottleneck is in infrastructure or client code, not the GPU.
@@ -0,0 +1,3 @@
+# Expert Parallelism
+ 
+Expert Parallelism distributes MoE experts across GPUs so each GPU hosts multiple complete experts. Tokens are routed to the GPU holding their activated experts, and each expert runs independently within a single GPU without splitting across multiple. EP has lower inter-GPU communication overhead than Tensor Parallelism and scales well to multi-node deployments via InfiniBand.
@@ -0,0 +1,3 @@
+# Few-Step Image Generation
+
+Few-step image generation models produce usable images in eight or fewer denoising steps by training the model to collapse the standard 30 to 50 step process into a fraction of the iterations. Techniques include latent consistency distillation and adversarial training, both of which teach the model to reach high-quality outputs in fewer passes. They are a strong fit for latency-sensitive use cases like real-time filters, but produce noticeably lower quality than full-step models and are a model selection decision rather than an inference optimization technique.
@@ -0,0 +1,3 @@
+# Fine-Tuning
+ 
+Fine-tuning takes a pretrained foundation model and adapts it to a specific domain by training on additional labeled data. A small fine-tuned model can match a much larger general model on constrained tasks like text-to-SQL. Smaller models are faster and cheaper to serve, so fine-tuning is one of the highest-leverage tools for hitting latency and cost targets.
@@ -0,0 +1,3 @@
+# FlashAttention
+ 
+FlashAttention is a series of hand-optimized CUDA kernels that compute attention with fewer reads and writes to GPU memory by fusing the attention algorithm into a single kernel. Standard attention stores intermediate matrices to memory between steps; FlashAttention eliminates those round-trips. FlashAttention 3 targets Hopper GPUs and FlashAttention 4 targets Blackwell GPUs.
@@ -0,0 +1,3 @@
+# Frameworks and Libraries
+ 
+Frameworks and libraries sit above CUDA and below inference engines, providing the abstractions that inference engineers interact with most directly. This group covers PyTorch and the Hugging Face ecosystem that supply the model implementations and tooling the rest of the stack depends on.
@@ -0,0 +1,3 @@
+# Geo-Aware Load Balancing
+ 
+A global load balancer routes requests to the GPU cluster geographically closest to the requesting user. Network latency accumulates at roughly 5 milliseconds per time zone traversed, so routing a request from Singapore to San Francisco adds 50 to 100 milliseconds of unnecessary overhead. Geo-aware routing ensures that spare capacity in a distant region is used only when nearby capacity is exhausted.
@@ -0,0 +1,3 @@
+# GPU Architecture
+ 
+GPU architecture determines the hard limits of what inference can achieve on a given device. This group covers compute, memory, interconnects, and sizing — the four dimensions an inference engineer needs to understand to make informed decisions about hardware selection, parallelism strategy, and optimization priorities.
@@ -0,0 +1,3 @@
+# GPU Models
+ 
+Knowing the GPU landscape — which devices are available, what they offer, and how they compare — is practical knowledge for procurement decisions, capacity planning, and understanding which optimizations apply to which hardware. This group covers the NVIDIA architectures that dominate inference deployments and the alternatives worth knowing.
@@ -0,0 +1,3 @@
+# GPU Procurement
+ 
+Cloud GPUs are available from hyperscalers like AWS and GCP, GPU-focused neoclouds like CoreWeave and Nebius, and secondary markets. Procurement mechanisms include reserved instances for predictable baseline load at discounted rates, on-demand instances for flexibility, and spot instances for cost-sensitive batch workloads that tolerate preemption. Large-scale deployments typically combine all three across multiple providers.
@@ -0,0 +1,3 @@
+# Grace and Vera CPUs
+ 
+NVIDIA's ARM-based Grace and Vera CPUs connect to their paired GPUs via NVLink Chip-to-Chip at up to 900 GB/s, far higher than standard PCIe connections. This high-bandwidth CPU-GPU link makes offloading KV caches and LoRA weights to CPU memory practical, since data can be retrieved quickly enough to remain on the critical path. Grace pairs with Hopper and Blackwell; Vera pairs with the upcoming Rubin architecture.
@@ -0,0 +1,3 @@
+# Hopper
+ 
+The Hopper architecture, released in 2022 with the H100 and later the H200, introduced FP8 Tensor Cores offering twice the compute throughput of FP16 and asynchronous data transfer features used by FlashAttention 3. H100 and H200 GPUs remain among the most widely deployed inference accelerators due to their maturity, broad software support, and relative availability compared to newer generations.
@@ -0,0 +1,3 @@
+# Image and Video Generation Bottlenecks
+ 
+Image and video generation models are compute-bound because each denoising step applies attention over the entire latent space in parallel, similar to LLM prefill. The models are also relatively small, so memory bandwidth is not the limiting factor. Optimizations focus on FLOPS efficiency rather than memory traffic reduction.
@@ -0,0 +1,3 @@
+# Image and Video Generation
+ 
+Image and video generation inference is compute-bound rather than memory-bound, making it a different optimization problem from LLM serving. This group covers the production patterns, kernel choices, and parallelism techniques specific to diffusion model deployment.
@@ -0,0 +1,3 @@
+# Image & Video Models
+
+Image and video models are specialized artificial intelligence architectures designed to process, interpret, and generate visual data. These models utilize techniques like convolutional neural networks (CNNs), transformers, and diffusion processes to analyze pixel patterns for tasks such as object detection, image classification, and video frame synthesis.
@@ -0,0 +1,3 @@
+# Image Generation Architecture
+ 
+Image generation models are pipelines of multiple components: a text encoder that converts the prompt into conditioning, a denoising model that iteratively refines a latent representation from noise into an image, and a variational autoencoder that converts the final latent to pixels. The denoising model is the largest and most compute-intensive component and runs up to 100 forward passes per image generation.
@@ -0,0 +1,3 @@
+# Image Generation Inference
+ 
+Production image generation inference uses SGLang Diffusion, TensorRT, or hand-optimized PyTorch. The most important optimization target is the attention kernel, where FlashAttention 3 or 4 outperforms the default on Hopper and Blackwell respectively. GEMM kernels for linear layers can be quantized to FP8 safely. Torch compilation with manual kernel plugins caches the compiled engine to avoid multi-minute recompilation on every cold start.
@@ -0,0 +1,3 @@
+# Independent Component Scaling
+ 
+Multi-model pipelines, like a voice agent that runs VAD, ASR, an LLM, and TTS in sequence, have components with different hardware requirements and traffic patterns. Each component should autoscale independently based on its own load. Running all components within the same Kubernetes cluster minimizes inter-component network latency, which can otherwise consume a significant fraction of a tight latency budget.
@@ -0,0 +1,3 @@
+# Inference Engines
+ 
+Inference engines are the core serving layer that takes model weights and handles requests at production scale. They implement continuous batching, memory management, parallelism, and optimization techniques so engineers do not have to implement these from scratch. This group covers the three dominant LLM engines and the orchestration platform that sits above them.
@@ -0,0 +1,3 @@
+# Inference vs. Training
+ 
+Training is the process of learning model weights from data. Inference is serving those weights in production to generate outputs. Training runs are compute-intensive batch jobs; inference is a continuous, latency-sensitive service. Where training success is measured in loss curves, inference success is measured in time to first token, tokens per second, and uptime.
@@ -0,0 +1,3 @@
+# Instances and Interconnects
+ 
+Within a node, NVLink connects GPUs one-to-one at up to 1800 GB/s on Blackwell, and NVSwitch creates an all-to-all fabric across all eight GPUs. Between nodes, InfiniBand provides inter-node connectivity at substantially lower bandwidth than NVLink. Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated slices, each with dedicated compute and memory, allowing small models to run on fractions of a high-end GPU without resource contention.
@@ -0,0 +1,3 @@
+# Inter-Token Latency (ITL)
+ 
+ITL is the time between successive generated tokens during decode. An ITL of 10 milliseconds corresponds to 100 tokens per second. It is the per-step measure underlying TPS and is directly tied to how quickly the GPU can complete a single decode forward pass.
@@ -0,0 +1,3 @@
+# Inference Engineer
+ 
+Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable to serve in production. It spans the full stack from CUDA kernels and GPU architecture to Kubernetes autoscaling and multi-cloud capacity management. The field emerged alongside the explosion of open models and the shift from pay-per-token APIs to dedicated deployments, where engineers own latency, throughput, cost, and uptime directly.
@@ -0,0 +1,3 @@
+# Kernel Selection and Fusion
+ 
+A CUDA kernel is a function that runs in parallel across thousands of GPU threads simultaneously. Kernel selection is choosing the best existing kernel for a specific operation; kernel fusion combines sequential kernels into one to eliminate unnecessary reads and writes to VRAM between operations. Libraries like cuBLAS, CUTLASS, CuTe, and FlashInfer provide pre-built kernels for common inference operations, and most inference engines perform kernel selection automatically.
@@ -0,0 +1,3 @@
+# KV Cache Storage
+ 
+KV caches can be stored in GPU VRAM (fastest, tens to hundreds of gigabytes), CPU RAM (slower, hundreds of gigabytes to terabytes), local SSD, or networked SSD. Moving less frequently accessed KV blocks to slower storage frees VRAM for active requests. NVIDIA Dynamo's KV Block Manager provides APIs for managing KV blocks across these storage tiers transparently.
@@ -0,0 +1,3 @@
+# KV Cache
+ 
+The KV cache stores the key and value tensors computed during attention for each previously processed token. Without it, attention would require recomputing every prior token on each decode step, making inference prohibitively slow. The KV cache lives in GPU memory and its size grows with sequence length and batch size, making it a major target for memory management techniques.
@@ -0,0 +1,3 @@
+# Large Language Models
+
+An LLM is a decoder-only transformer with three main components: an embedding layer that converts input tokens to vectors, a stack of transformer blocks that process those vectors, and an output language modeling head that converts the final hidden states into a probability distribution over the vocabulary. The configuration file for an open model specifies all architectural details including layer count, hidden dimensions, and attention heads.
@@ -0,0 +1,3 @@
+# Latency Percentiles
+ 
+Mean latency hides outliers. LLM response times follow a right-skewed distribution, so engineers measure P50, P90, P95, and P99. A P99 of several seconds means one in a hundred users has a poor experience even if the median is fast. Reducing P90/P99 latency is as important as reducing average latency.
@@ -0,0 +1,3 @@
+# Latency vs. Throughput
+ 
+Lower latency means faster responses per user; higher throughput means more total requests served per unit time. These goals are often in tension. Increasing batch size improves throughput but worsens per-user latency. Choosing the right balance is one of the primary decisions in configuring an inference service.
@@ -0,0 +1,3 @@
+# LLM-based Models
+
+LLM-based embedding models use a decoder-only transformer backbone, typically between one and eight billion parameters, to produce higher-quality semantic representations than BERT-style models. They apply a pooling operation to the final layer hidden states and are trained with contrastive objectives on large-scale retrieval datasets. The quality advantage is most pronounced on complex or cross-lingual retrieval tasks; the cost is higher compute per embedding and the need for GPU inference rather than CPU.
@@ -0,0 +1,3 @@
+# LLM Inference Bottlenecks
+ 
+LLM prefill is compute-bound because it processes the entire input sequence as one large matrix multiplication, performing many operations per byte loaded. LLM decode is memory-bound because it generates tokens one at a time via vector-matrix multiplications that require loading all model weights from memory for each token while performing relatively few arithmetic operations.
@@ -0,0 +1,3 @@
+# Local and Edge Inference
+ 
+Not all inference runs in a datacenter. Desktop and mobile deployments have different hardware constraints, tooling ecosystems, and use cases from cloud inference. This group covers the patterns and tools for running models on consumer hardware and mobile devices.
@@ -0,0 +1,3 @@
+# Locust
+
+Locust is a general-purpose open-source load testing framework written in Python. In inference benchmarking it is used to simulate realistic user traffic patterns — including request arrival rates, concurrency ramps, and burst scenarios — that synthetic tools do not capture. It is particularly useful for end-to-end load testing that includes network overhead, queuing, and autoscaling behavior rather than raw model server performance alone.
@@ -0,0 +1,3 @@
+# Long Context Handling
+ 
+Long sequences produce KV caches large enough to strain GPU memory, slow down decode, and increase prefill time. Techniques for handling long context include FlashAttention for memory-efficient attention computation, PagedAttention for non-contiguous KV storage, chunked prefill to interleave long prefill with decode, and model-level approaches like sliding window attention that limit how many prior tokens each token attends to.
@@ -0,0 +1,3 @@
+# LoRA-Aware Routing
+ 
+LoRA fine-tune weights are small adapter layers loaded on top of a base model to specialize its behavior for a specific task or customer. A deployment serving many LoRA variants for the same base model must route each request to a replica that already has the required LoRA loaded in GPU memory. Routing without this awareness forces constant LoRA swapping, adding latency and reducing throughput on each affected replica.
@@ -0,0 +1,3 @@
+# Measuring Quality Impact
+ 
+After quantizing a model, quality must be validated against the original. Three methods are used: perplexity scoring (lower is better, with minimal increase after quantization), standard intelligence benchmarks like MMLU or SWE-bench, and product-specific evals tailored to the actual use case. All three should show differences indistinguishable from noise before a quantized model is deployed.
@@ -0,0 +1,3 @@
+# Memory
+ 
+GPU memory determines what fits on a single device and how fast data can move through the compute units. VRAM capacity sets the upper limit on model size and KV cache; VRAM bandwidth determines how quickly the decode phase can load weights per token. Modern datacenter GPUs use high-bandwidth memory (HBM) in generations HBM3, HBM3e, and HBM4, stacked directly on the GPU die to maximize bandwidth.
@@ -0,0 +1,3 @@
+# Mixture of Experts (MoE)
+ 
+Mixture of Experts models replace dense linear layers with a set of smaller expert matrices, activating only a subset of experts per token via a learned router. A model like Qwen3-235B activates only 22 billion of its 235 billion parameters per forward pass, making single-user inference efficient. In batched production serving, different tokens activate different experts, so nearly all parameters are active across the batch.
@@ -0,0 +1,3 @@
+# Mobile Inference
+ 
+Mobile devices run inference using vendor frameworks: Google's AI Edge SDK for Android and Apple's Foundation Models and Core ML for iOS. Hardware limitations restrict practical model sizes to one to two billion parameters on current phones. Small fine-tuned models for tasks like transcription, translation, and on-device classification are the strongest fit for mobile edge inference.
@@ -0,0 +1,3 @@
+# Model Evaluation
+ 
+Model evaluation, or evals, is the practice of systematically measuring model quality for a specific task or domain. Standard intelligence benchmarks like MMLU measure general capabilities but can be gamed; product-specific evals that test real use cases provide more reliable signal. High-confidence evals are a prerequisite before investing in inference optimization, ensuring the model is actually useful before making it fast.
@@ -0,0 +1,3 @@
+### Model Formats and Runtimes
+ 
+Before a model can serve requests, its weights must be stored, loaded, and compiled into an executable form. This group covers the file formats used to distribute weights and the runtimes that compile them into optimized inference engines for specific hardware targets.
@@ -0,0 +1,3 @@
+# Model Parallelism
+ 
+When a model is too large to fit on a single GPU, or when serving latency requires more compute than one device can provide, model parallelism distributes the workload across multiple GPUs or nodes. This group covers the three parallelism strategies and their trade-offs in terms of communication overhead, latency, and hardware requirements.
@@ -0,0 +1,3 @@
+# Model Selection
+ 
+The most important decision in model performance optimization is which model to use. Smaller models with fewer parameters are faster and cheaper to serve than larger ones. The goal is to find the smallest model that meets quality requirements for the task. Model architecture also determines which inference engines and optimization techniques are available, so popular architectures with broad tooling support are preferable.
@@ -0,0 +1,3 @@
+# ModelOpt
+ 
+ModelOpt is NVIDIA's toolkit for post-training quantization, supporting FP8, INT8, FP4, MXFP8, MXFP4, and NVFP4 output formats. It integrates with PyTorch and exports quantized models directly to TensorRT-LLM, vLLM, and SGLang. Calibration runs a small set of representative inputs through the model to compute per-layer scale factors that minimize the gap between quantized and original outputs.
@@ -0,0 +1,3 @@
+#  Multi-Cloud
+ 
+Multi-cloud inference distributes GPU capacity across providers and regions to maximize availability, minimize latency, and meet data residency requirements. This group covers procurement strategy, capacity management, geographic routing, and the reliability patterns that make large-scale inference resilient to hardware and provider failures.
@@ -0,0 +1,3 @@
+# Multi-Node Inference
+ 
+Multi-node inference spreads a model across more than eight GPUs using InfiniBand for inter-node communication. It is necessary for the largest models where a single node lacks sufficient VRAM for weights and KV cache. InfiniBand bandwidth is an order of magnitude lower than NVLink, so parallelism strategies must minimize cross-node communication. For MoE models, Expert Parallelism is preferred over Tensor Parallelism between nodes.
@@ -0,0 +1,3 @@
+# N-gram Speculation
+ 
+N-gram speculation builds a dictionary during prefill that maps token prefixes to observed n-gram continuations from the input. During decode, generated tokens are looked up in this dictionary and matching suffixes become draft tokens with no separate model required. It is most effective for code completion, where output frequently mirrors input syntax, and can produce very long draft sequences with high acceptance rates in that domain.
@@ -0,0 +1,3 @@
+# Neural Networks
+ 
+A neural network is a layered structure of nodes where each node applies a learned weight matrix and bias to its input. Multiple layers connected in sequence form the foundation for all generative AI models. Inference engineers need a working model of how neural networks compute outputs to understand where performance bottlenecks arise.
@@ -0,0 +1,3 @@
+# NIMs
+ 
+NIMs are pre-built Docker containers provided by NVIDIA for popular open models, available as flexible multi-LLM containers or as GPU- and model-specific optimized containers. They provide a ready-made starting point for serving common architectures without building a custom container from scratch. For maximum control and custom configurations, starting from a less opinionated base image and building up is generally preferable.
@@ -0,0 +1,3 @@
+# NVIDIA Dynamo
+ 
+Dynamo supports conditional disaggregation, where requests are first sent to the decode engine and only routed to a prefill engine when the input is long or not already cached. The ratio of prefill to decode engines is configurable at runtime and can shift as traffic patterns change. This dynamic allocation prevents the prefill queue from becoming a bottleneck and avoids wasting decode engine resources on prefill-heavy traffic.
@@ -0,0 +1,3 @@
+# NVIDIA Dynamo
+ 
+Dynamo is NVIDIA's open-source distributed serving platform that sits above inference engines and provides orchestration for large-scale deployments. It adds KV-aware routing, disaggregated prefill/decode serving, multi-node parallelism, and an SLA-based planner that dynamically adjusts the number of prefill and decode workers based on TTFT and TPS targets.
@@ -0,0 +1,3 @@
+# NVIDIA GenAI-Perf
+
+GenAI-Perf is NVIDIA's inference benchmarking tool, designed to work across TensorRT-LLM, Triton Inference Server, and OpenAI-compatible endpoints. It generates synthetic workloads with configurable sequence length distributions and concurrency levels and outputs standard latency and throughput metrics including TTFT, ITL, and TPS percentiles.
@@ -0,0 +1,3 @@
+# NVIDIA
+ 
+NVIDIA GPUs are the dominant hardware for inference due to their combination of Tensor Core compute, HBM bandwidth, NVLink interconnects, and the CUDA software ecosystem. The three most relevant architectures for current inference deployments are Hopper, Blackwell, and the Grace and Vera CPU platforms that pair with them.
@@ -0,0 +1,3 @@
+# Observability
+ 
+Inference observability tracks total request volume, input and output sequence lengths, response codes, latency percentiles (P50, P90, P99 of TTFT, TPS, and end-to-end latency), replica count, GPU utilization, and queue depth. These metrics are interdependent; a latency spike could come from long input sequences, high concurrency, or a hardware issue. Observability must integrate with existing tooling like Grafana, Datadog, and PagerDuty to put inference metrics in context.
@@ -0,0 +1,3 @@
+# Omni-Modal Models
+ 
+Omni models accept multiple input modalities and produce multiple output types in a single model. They avoid the latency of running separate specialized models in sequence but often lag behind dedicated models in quality within each modality. Production pipelines using VLMs typically coordinate preprocessing steps like OCR, audio transcription, and PDF extraction as separate independently scaling components.
@@ -0,0 +1,3 @@
+# Online vs. Offline Inference
+ 
+Online inference serves real-time user-facing requests where latency matters. Offline inference processes large batches of data asynchronously where throughput and cost matter more than per-request speed. The same model, like Whisper for transcription, may need separate deployments for each use case because the optimal configurations differ significantly.
@@ -0,0 +1,3 @@
+# ONNX Runtime
+ 
+ONNX Runtime is an open-source inference runtime that executes models exported to the ONNX format, which stores both weights and a computation graph for cross-platform portability. It supports multiple hardware backends including CUDA, DirectML, and CPU. It is widely used for embedding models, vision models, and pipelines that need to run across hardware environments without engine-specific configuration.
@@ -0,0 +1,3 @@
+# Open vs. Closed Models
+ 
+Closed models are accessed exclusively through a vendor API; the weights are never released. Open models publish their weights for download and self-hosting. For inference engineers, this distinction determines the entire problem space: closed models require no infrastructure but offer no control over latency, cost, or configuration, while open models require building and operating a full inference stack but give complete control over every layer of it.
@@ -0,0 +1,3 @@
+# Ops:Byte Ratio
+
+The ops:byte ratio is a property of GPU hardware that expresses the balance between compute throughput and memory bandwidth. It is calculated by dividing peak FLOPS by peak memory bandwidth, and sets the threshold that separates compute-bound from memory-bound workloads on that device. Different GPU generations have very different ops:byte ratios, which is why an optimization that works well on one generation may have no effect on another.
@@ -0,0 +1,3 @@
+# Other Accelerators
+ 
+Alternatives to NVIDIA GPUs include AMD MI350, Google TPUs, AWS Inferentia and Trainium, Cerebras WSE-3, and Groq LPUs. Each offers a specific advantage: AMD provides CUDA-compatible tooling via ROCm; TPUs integrate deeply with Google Cloud and JAX; Inferentia and Trainium offer cost advantages within AWS. All face the shared challenge of building an ecosystem to match the breadth of NVIDIA's CUDA libraries and inference engine support.
@@ -0,0 +1,3 @@
+# PagedAttention
+ 
+PagedAttention stores the KV cache in fixed-size pages rather than a single contiguous memory block, allowing it to be distributed across GPU memory without fragmentation. It uses a lookup table to access pages on demand, reducing memory waste from over-allocation and enabling more requests to share GPU memory. It is the memory management technique that made vLLM's high throughput possible.
@@ -0,0 +1,3 @@
+# Performance Benchmarking
+ 
+Benchmarking measures inference service performance under realistic workloads. Good benchmarks simulate realistic input and output sequence lengths, concurrency patterns, and request contents rather than synthetic uniform inputs. Standard tools include SGLang Genai-bench, NVIDIA GenAI-Perf, and Locust. Always establish a baseline before applying optimizations and change one variable at a time.
@@ -0,0 +1,3 @@
+# Pipeline Parallelism
+ 
+Pipeline Parallelism splits an LLM's layers into sequential stages and assigns each stage to a different GPU or node. Unlike Tensor Parallelism, each GPU holds complete layers rather than layer fragments. It introduces pipeline bubbles where GPUs wait for preceding stages, reducing efficiency. It is used for multi-node inference with dense models, typically combined with Tensor Parallelism within each node.
@@ -0,0 +1,3 @@
+Post-Training Quantization
+
+Post-training quantization converts finished model weights to lower precision after training is complete, using a small calibration dataset to compute per-layer scale factors that minimize the gap between quantized and original outputs. It is the standard approach for most deployments since it requires no retraining and can be applied to any existing model. The leading tool is NVIDIA ModelOpt, which exports quantized models in formats compatible with all major inference engines.
--- a/Show More
+++ b/Show More