From 56eff6dd97ab2a4a7cb0fdc5a91adfbf4e45c760 Mon Sep 17 00:00:00 2001 From: Javier Canales Date: Thu, 1 Oct 2026 13:37:34 +0200 Subject: [PATCH] feat(inference-engineering): reuse AI Engineer roadmap resources Add 39 resources already curated on the AI Engineer roadmap to 22 Inference Engineering topics where they fit (model selection and evaluation, latency/cost trade-offs, caching, long context, embeddings, ASR, VLMs, diffusion, desktop inference). At most 4 new links per topic; existing links are kept and everything stays type-ordered. Co-Authored-By: Claude Opus 5.5 --- .../auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md | 2 ++ .../content/bert-style-models@ef3EneMibOaDQkKcpipGY.md | 2 ++ .../content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md | 4 +++- .../content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md | 6 +++++- .../content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md | 6 +++++- .../content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md | 2 ++ .../content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md | 4 +++- .../content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md | 4 +++- .../content/fine-tuning@fWJplrAuUStEMNNLQc75G.md | 4 +++- .../content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md | 4 +++- .../content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md | 3 +++ .../content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md | 6 +++++- .../content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md | 5 ++++- .../content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md | 3 +++ .../content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md | 2 ++ .../content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md | 2 ++ .../content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md | 6 +++++- .../shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md | 7 ++++++- .../streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md | 6 +++++- .../transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md | 3 ++- .../video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md | 6 +++++- .../vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md | 3 ++- 22 files changed, 75 insertions(+), 15 deletions(-) diff --git a/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md b/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md index db87f0d5a..6df8f2e60 100644 --- a/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md +++ b/roadmaps/inference-engineering/content/auto-speech-recognition@6PRICRK6aRQCoThxnevB0.md @@ -5,4 +5,6 @@ ASR models like Whisper take audio input and produce text transcripts. Whisper i Visit the following resources to learn more: - [@official@OpenAI Whisper Models on Hugging Face](https://huggingface.co/openai) +- [@official@OpenAI Whisper](https://openai.com/index/whisper/) +- [@opensource@openai/whisper](https://github.com/openai/whisper) - [@article@Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)](https://arxiv.org/abs/2212.04356) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md b/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md index 0e2705e90..85187b641 100644 --- a/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md +++ b/roadmaps/inference-engineering/content/bert-style-models@ef3EneMibOaDQkKcpipGY.md @@ -4,5 +4,7 @@ BERT-style encoder-only models are the traditional architecture for text embeddi Visit the following resources to learn more: +- [@official@SentenceTransformers Documentation](https://sbert.net/) +- [@official@Using Sentence Transformers at Hugging Face](https://huggingface.co/docs/hub/sentence-transformers) - [@article@BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) - [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md b/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md index 10959d4d9..9b2a9f2d8 100644 --- a/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md +++ b/roadmaps/inference-engineering/content/caching-approaches@2nGv7_8PTWYr7uqbudCcX.md @@ -5,4 +5,6 @@ KV cache reuse is the primary caching strategy in LLM inference, covering both p Visit the following resources to learn more: - [@opensource@LMCache](https://github.com/LMCache/LMCache) -- [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444) \ No newline at end of file +- [@article@CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444) +- [@article@What is Prompt Caching?](https://www.ibm.com/think/topics/prompt-caching) +- [@video@What is Prompt Caching? Optimize LLM Latency with AI Transformers](https://www.youtube.com/watch?v=u57EnkQaUTY) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md b/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md index baf437097..10687e429 100644 --- a/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md +++ b/roadmaps/inference-engineering/content/client-integration@3RkQpvRGWh1Hp_qHlTiXX.md @@ -1,3 +1,7 @@ # Client Integration -The client layer connects application code to the inference service and contributes its own slice of end-to-end latency. This group covers the connection overhead, async request patterns, and protocol choices that determine how efficiently client and server communicate. \ No newline at end of file +The client layer connects application code to the inference service and contributes its own slice of end-to-end latency. This group covers the connection overhead, async request patterns, and protocol choices that determine how efficiently client and server communicate. + +Visit the following resources to learn more: + +- [@article@OpenAI-Compatible API](https://bentoml.com/llm/llm-inference-basics/openai-compatible-api) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md b/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md index eb3eda9fd..dfab33158 100644 --- a/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md +++ b/roadmaps/inference-engineering/content/cost-estimation@Gof4kW2LbkNCtH68c7Xvq.md @@ -1,3 +1,7 @@ # Cost Estimation -Dedicated inference cost depends on batch sizing, traffic patterns, and sequence lengths rather than simply on token count. Comparing dedicated versus API costs requires converting per-token API pricing into total cost over a representative time window and comparing it to GPU-hour costs at the same traffic volume. Total cost of ownership also includes engineering time for building and maintaining the inference system. \ No newline at end of file +Dedicated inference cost depends on batch sizing, traffic patterns, and sequence lengths rather than simply on token count. Comparing dedicated versus API costs requires converting per-token API pricing into total cost over a representative time window and comparing it to GPU-hour costs at the same traffic volume. Total cost of ownership also includes engineering time for building and maintaining the inference system. + +Visit the following resources to learn more: + +- [@article@The LLM Inference Trilemma: Throughput, Latency, Cost](https://www.digitalocean.com/blog/llm-inference-tradeoffs) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md index c309427bc..52e3ddc9d 100644 --- a/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md +++ b/roadmaps/inference-engineering/content/desktop-inference@fQDl7_3nRGa8NFivOIgTP.md @@ -4,4 +4,6 @@ Desktop inference runs on consumer hardware, primarily Apple M-series unified me Visit the following resources to learn more: +- [@official@Ollama](https://ollama.com/) +- [@official@LM Studio](https://lmstudio.ai/) - [@opensource@ComfyUI](https://github.com/comfyanonymous/ComfyUI) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md index d9b7a49a2..6e6aea720 100644 --- a/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md +++ b/roadmaps/inference-engineering/content/diffusion-models@aagMwPPmi6ngfkIqZL-Mg.md @@ -7,4 +7,6 @@ Visit the following resources to learn more: - [@article@Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) - [@article@High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) - [@article@SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) -- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) \ No newline at end of file +- [@article@Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) +- [@article@How DALL-E 2 Actually Works](https://www.assemblyai.com/blog/how-dall-e-2-actually-works/) +- [@video@How AI Image Generators Work (Stable Diffusion / DALL-E)](https://www.youtube.com/watch?v=1CIpzeNxIhU) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md b/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md index 5e057244f..5b1a764b4 100644 --- a/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md +++ b/roadmaps/inference-engineering/content/embedding-models@Tvt-vX1gWuJZJBHh5vfKu.md @@ -4,7 +4,9 @@ Embedding models convert variable-length text or image input into a fixed-length Visit the following resources to learn more: +- [@official@Hugging Face Embedding Models](https://huggingface.co/models?pipeline_tag=feature-extraction) - [@article@BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) - [@article@Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084) - [@article@Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147) -- [@article@MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316) \ No newline at end of file +- [@article@MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316) +- [@article@What Are Embedding Models? Benefits and Best Practices](https://cohere.com/blog/embedding-models) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md b/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md index 15d97e342..36e0bafba 100644 --- a/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md +++ b/roadmaps/inference-engineering/content/fine-tuning@fWJplrAuUStEMNNLQc75G.md @@ -4,4 +4,6 @@ Fine-tuning takes a pretrained foundation model and adapts it to a specific doma Visit the following resources to learn more: -- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) \ No newline at end of file +- [@article@Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) +- [@article@What is Fine-Tuning?](https://www.ibm.com/think/topics/fine-tuning) +- [@article@How I Fine-Tuned Granite-Vision 2B to Beat a 90B Model](https://towardsdatascience.com/how-i-fine-tuned-granite-vision-2b-to-beat-a-90b-model-insights-and-lessons-learned/?utm_source=roadmap&utm_medium=Referral&utm_campaign=TDS+roadmap+integration) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md b/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md index 7fc1c4c8b..c44bd14e3 100644 --- a/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md +++ b/roadmaps/inference-engineering/content/inference-vs-training@fVUfQYb8BP_hFMmd1S2-J.md @@ -4,4 +4,6 @@ Training is the process of learning model weights from data. Inference is servin Visit the following resources to learn more: -- [@article@Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165) \ No newline at end of file +- [@article@Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165) +- [@article@Inference vs Training](https://www.cloudflare.com/learning/ai/inference-vs-training/) +- [@article@What is Machine Learning Inference?](https://hazelcast.com/glossary/machine-learning-inference/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md index 9092ef62d..70196dbe5 100644 --- a/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md +++ b/roadmaps/inference-engineering/content/large-language-models@irgGvEnY1_WW1EPtuMKgC.md @@ -4,4 +4,7 @@ An LLM is a decoder-only transformer with three main components: an embedding la Visit the following resources to learn more: +- [@article@What is a Large Language Model (LLM)?](https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/) +- [@article@New to LLMs? Start Here](https://towardsdatascience.com/new-to-llms-start-here/?utm_source=roadmap&utm_medium=Referral&utm_campaign=TDS+roadmap+integration) +- [@video@How Large Language Models Work](https://www.youtube.com/watch?v=5sLYAQS9sWQ) - [@book@Build a Large Language Model From Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md b/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md index a8856c0e2..d769b26c5 100644 --- a/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md +++ b/roadmaps/inference-engineering/content/latency-vs-throughput@kvNIi3aY5rn69_hYkDyUK.md @@ -1,3 +1,7 @@ # Latency vs. Throughput -Lower latency means faster responses per user; higher throughput means more total requests served per unit time. These goals are often in tension. Increasing batch size improves throughput but worsens per-user latency. Choosing the right balance is one of the primary decisions in configuring an inference service. \ No newline at end of file +Lower latency means faster responses per user; higher throughput means more total requests served per unit time. These goals are often in tension. Increasing batch size improves throughput but worsens per-user latency. Choosing the right balance is one of the primary decisions in configuring an inference service. + +Visit the following resources to learn more: + +- [@article@The LLM Inference Trilemma: Throughput, Latency, Cost](https://www.digitalocean.com/blog/llm-inference-tradeoffs) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md b/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md index dc5aba8c7..14ce8aea2 100644 --- a/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md +++ b/roadmaps/inference-engineering/content/long-context-handling@uL3Fn8XfS1Yv_PJd9R8wi.md @@ -5,4 +5,7 @@ Long sequences produce KV caches large enough to strain GPU memory, slow down de Visit the following resources to learn more: - [@article@Ring Attention with Blockwise Transformers for Near-Infinite Context](https://arxiv.org/abs/2310.01889) -- [@article@RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864) \ No newline at end of file +- [@article@RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864) +- [@article@What is Long Context and Why Does it Matter for AI?](https://cloud.google.com/transform/the-prompt-what-are-long-context-windows-and-why-do-they-matter) +- [@article@Long-Context LLM Infrastructure: Building Systems for Million-Token Windows](https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide) +- [@video@Deep Dive into Long Context](https://www.youtube.com/watch?v=NHMJ9mqKeMQ) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md b/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md index 9fbd6f1b5..ec66fcb23 100644 --- a/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md +++ b/roadmaps/inference-engineering/content/model-evaluation@lVf5GoJF_PVTkTiQFTNVg.md @@ -5,4 +5,7 @@ Model evaluation, or evals, is the practice of systematically measuring model qu Visit the following resources to learn more: - [@article@How to Fine-Tune Qwen3 to GPT-4o Level Performance](https://ghost.oxen.ai/how-to-fine-tune-qwen3-to-gpt-4o-level-performance/) +- [@article@LLM Evaluation: A Beginner's Guide](https://www.evidentlyai.com/llm-guide/llm-evaluation) +- [@article@LLM Evaluation Metrics and Methods, Explained Simply](https://www.evidentlyai.com/llm-guide/llm-evaluation-metrics) +- [@article@A Pragmatic Guide to LLM Evals for Devs](https://newsletter.pragmaticengineer.com/p/evals) - [@book@Evals for AI Engineers](https://www.oreilly.com/library/view/evals-for-ai/9798341660717/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md b/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md index 4d05c2e9a..260ab4ff6 100644 --- a/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md +++ b/roadmaps/inference-engineering/content/model-selection@jQAWZ1plw4_wPE1UvE_Ux.md @@ -4,4 +4,6 @@ The most important decision in model performance optimization is which model to Visit the following resources to learn more: +- [@article@Choosing the Right Model](https://bentoml.com/llm/getting-started/choosing-the-right-model) +- [@article@Beyond Vibes: How to Properly Select the Right LLM for the Right Task](https://aws.amazon.com/blogs/machine-learning/beyond-vibes-how-to-properly-select-the-right-llm-for-the-right-task/) - [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md b/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md index 0c7606d17..cddb72aba 100644 --- a/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md +++ b/roadmaps/inference-engineering/content/open-vs-closed-models@KAt5FzglOUtH7jLlzDn-I.md @@ -4,4 +4,6 @@ Closed models are accessed exclusively through a vendor API; the weights are nev Visit the following resources to learn more: +- [@article@Open-Source vs. Closed LLMs: A Guide](https://hatchworks.com/blog/gen-ai/open-source-vs-closed-llms-guide/) +- [@video@Open Source vs Closed AI: LLMs, Agents & the AI Stack Explained](https://www.youtube.com/watch?v=_QfxGZGITGw) - [@book@AI Engineering: Building Applications with Foundation Models](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md b/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md index 5b7293769..6f791376d 100644 --- a/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md +++ b/roadmaps/inference-engineering/content/reasoning-models@Y9YkcfblXHEvS-gdEWwaE.md @@ -1,3 +1,7 @@ # Reasoning Models -Reasoning models generate an intermediate thinking sequence before producing their final output, adding a third token sequence alongside the input and output. This reasoning trace increases total token count significantly and changes inference economics: TTFT is higher, total latency is longer, and output token costs dominate. Inference engineers must account for the variable and sometimes very long reasoning sequences when sizing infrastructure and setting latency budgets. \ No newline at end of file +Reasoning models generate an intermediate thinking sequence before producing their final output, adding a third token sequence alongside the input and output. This reasoning trace increases total token count significantly and changes inference economics: TTFT is higher, total latency is longer, and output token costs dominate. Inference engineers must account for the variable and sometimes very long reasoning sequences when sizing infrastructure and setting latency budgets. + +Visit the following resources to learn more: + +- [@article@Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md b/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md index 0feb4dce3..ae95e3991 100644 --- a/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md +++ b/roadmaps/inference-engineering/content/shared-vs-dedicated-inf@2DsjfW9r4dOI3LDQU80qt.md @@ -1,3 +1,8 @@ # Shared vs. Dedicated Inference -Shared inference means sending requests to a public API and paying per token. Dedicated inference means renting GPUs and running your own model server, paying per GPU-hour. Shared inference is simpler and has no cold start, but dedicated inference offers control over latency, cost at scale, model customization, and uptime. Most products start shared and move to dedicated as traffic grows. \ No newline at end of file +Shared inference means sending requests to a public API and paying per token. Dedicated inference means renting GPUs and running your own model server, paying per GPU-hour. Shared inference is simpler and has no cold start, but dedicated inference offers control over latency, cost at scale, model customization, and uptime. Most products start shared and move to dedicated as traffic grows. + +Visit the following resources to learn more: + +- [@article@Serverless vs. Self-Hosted LLM Inference](https://bentoml.com/llm/llm-inference-basics/serverless-vs-self-hosted-llm-inference) +- [@article@Self-Hosted LLM: A Practical Guide for DevOps](https://www.plural.sh/blog/self-hosting-large-language-models/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md b/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md index b4bdafea8..097129910 100644 --- a/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md +++ b/roadmaps/inference-engineering/content/streaming--protocol-support@SGQCvuHAkg4TcJa_e2f-y.md @@ -1,3 +1,7 @@ # Streaming and Protocol Support -Text streaming over HTTP is sufficient for LLM chat applications. For real-time voice and video applications, WebSockets provide a persistent bidirectional connection for continuous unstructured data like audio chunks. gRPC provides bidirectional streaming for structured service-to-service communication with schema enforcement. Each WebSocket connection holds a slot on the server, and the concurrency limit must be set to match the GPU's sustainable load. \ No newline at end of file +Text streaming over HTTP is sufficient for LLM chat applications. For real-time voice and video applications, WebSockets provide a persistent bidirectional connection for continuous unstructured data like audio chunks. gRPC provides bidirectional streaming for structured service-to-service communication with schema enforcement. Each WebSocket connection holds a slot on the server, and the concurrency limit must be set to match the GPU's sustainable load. + +Visit the following resources to learn more: + +- [@article@AI for Web Devs: Faster Responses with HTTP Streaming](https://austingil.com/ai-for-web-devs-streaming/) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md b/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md index 56e1f2bb5..a54e5c4e4 100644 --- a/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md +++ b/roadmaps/inference-engineering/content/transformers--diffusers@H_e3KZ7p7enUCYE5M5Lr7.md @@ -5,4 +5,5 @@ The transformers and diffusers libraries provide reference implementations of po Visit the following resources to learn more: - [@official@Hugging Face Diffusers](https://huggingface.co/docs/diffusers/index) -- [@official@Hugging Face Transformers](https://huggingface.co/docs/transformers/index) \ No newline at end of file +- [@official@Hugging Face Transformers](https://huggingface.co/docs/transformers/index) +- [@course@Hugging Face Course](https://www.youtube.com/playlist?list=PLo2EIpI_JMQvWfQndUesu0nPBAtZ9gP1o) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md b/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md index ec7d9f600..54ea63650 100644 --- a/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md +++ b/roadmaps/inference-engineering/content/video-processing-for-vlms@7pRZu7xa2_43HVwKty9px.md @@ -1,3 +1,7 @@ # Video Processing for VLMs -One second of video contains 24 frames, each requiring roughly 1,000 tokens to represent at full resolution. This makes video inputs extremely long-context workloads. Frame rate reduction and downsampling are practically mandatory for running video through a VLM in a single request. Prefix caching and chunked prefill become critical for managing the resulting multi-thousand-token sequences. \ No newline at end of file +One second of video contains 24 frames, each requiring roughly 1,000 tokens to represent at full resolution. This makes video inputs extremely long-context workloads. Frame rate reduction and downsampling are practically mandatory for running video through a VLM in a single request. Prefix caching and chunked prefill become critical for managing the resulting multi-thousand-token sequences. + +Visit the following resources to learn more: + +- [@opensource@Awesome LLMs for Video Understanding](https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md b/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md index e0c34469a..e471144b0 100644 --- a/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md +++ b/roadmaps/inference-engineering/content/vision-language-models@JsjmQrN1V_dM1Kw6UnYLc.md @@ -7,4 +7,5 @@ Visit the following resources to learn more: - [@article@BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597) - [@article@Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) - [@article@Segment Anything](https://arxiv.org/abs/2304.02643) -- [@article@SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://arxiv.org/abs/2509.11815) \ No newline at end of file +- [@article@SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://arxiv.org/abs/2509.11815) +- [@course@A Multimodal World - Hugging Face Computer Vision Course](https://huggingface.co/learn/computer-vision-course/en/unit4/multimodal-models/a_multimodal_world) \ No newline at end of file