←搜索结果

搜索“NVIDIA”

按匹配度 · 共 91 条结果 清除搜索

星期四

  1. Mistral AI精选

    堆也会撒谎：调试 vLLM 内存泄漏

    Mistral AI 定位了 vLLM 在 Mistral Medium 3.1、图编译及 Prefill/Decode 分离部署下每分钟增长 400 MB 的内存泄漏，根因是 NIXL 依赖的 UCX 拦截 mmap，并让待清理区域在 RCache 失效队列中累积。设置 UCX_MEM_MMAP_HOOK_MODE=none 可消除泄漏且未影响性能，另可将 UCX_RCACHE_MAX_UNRELEASED 设为 1024；相关修复已合并至 vLLM，UCX 与 NIXL 团队也计划在未来版本中调整行为。

    E2f0fa05 5d77 4acc Ac9b 43e10db5769c Copie

    tance and collaboration in resolving this issue: Robert Shaw (Red Hat, vLLM and llm-d Maintainer) Will Eaton (Red Hat, vLLM and llm-d Maintainer) Nicolò Lucchesi (Red Hat, vLLM and llm-d Maintainer) Mikhail Brinskii (NVIDIA, NIXL Maintainer) Leonid Genkin (NVIDIA, UCX Maintainer) Interested in tackling challenges like these? Join us at Mistral AI and help shape the future of AI infrastructure. We’re always looking for talented engineers and researchers to collaborate on cutting-edge projects. 0%

    匹配字段正文

    #lab#教程/实践#部署/工程#开源生态

星期三

  1. Ahead of AI精选

    DeepSeek V3 至 V3.2 模型技术演进全览

    DeepSeek V3.2 延续 V3 系列架构，并引入 DeepSeek Sparse Attention、自验证、自改进及多项 GRPO 训练调整。文章梳理了 V3、R1、V3.1、V3.2-Exp、DeepSeekMath V2 和 V3.2-Speciale 的演进，解释 MLA、RLVR 与混合推理模式等关键机制。更新部分还介绍了用于改善训练稳定性和收敛表现的 mHC 残差路径研究。

    saw on various social media platforms that people were pronouncing the team “dead” (as a one-hit wonder). I am sure the DeepSeek team has also been busy navigating the switch from NVIDIA to Huawei chips. By the way, I am not affiliated with them or have spoken with them; everything here is based on public information. As far as I know, they are back to using NVIDIA chips. Finally, it’s also not that they haven’t released anything. There have been a couple of smaller releases that trickled in thi

    匹配字段正文

    #newsletter#教程/实践#推理#数据/训练

星期二

  1. Ahead of AI

    标准 LLM 之外有哪些替代架构

    Sebastian Raschka 梳理了标准自回归 Transformer 之外的线性注意力混合架构、文本扩散模型、代码世界模型和小型递归 Transformer。文章比较了这些路线在长上下文效率、生成质量、代码建模和专用推理上的取舍，并认为通用项目目前仍应优先采用成熟的自回归 Transformer。

    transformer components, whereas I see the models discussed here (Qwen3-Next and Kimi Linear) as transformers with SSM components. However, since I have listed IBM Granite 4.0 and NVIDIA Nemotron Nano 2 in the transformer-SSM box, an argument could be made for putting them into a single category.) If you are working with or on LLMs, for example, building applications, fine-tuning models, or trying new algorithms, I would make these models my go-to. They are tested, proven, and perform well. More

    匹配字段正文

    #newsletter#教程/实践#推理#编码
  2. Modal 官方工程博客

    Modal 用 Pipecat 和开源模型实现一秒内语音到语音延迟

    Modal 工程博客介绍了如何用 Modal、开源框架 Pipecat 和开源权重模型搭建语音 AI 聊天机器人，在客户端与容器就近部署时实现约一秒的中位语音到语音延迟。方案采用 Parakeet-tdt-v3 做语音转文字、Qwen3-4B-Instruct-2507 配合 vLLM 做 LLM 推理、Kokoro 做语音合成，并用 ChromaDB 与 all-minilm-l6-v2 完成 RAG 检索。作者还分享了用 Modal Tunnels 绕过输入平面降低网络延迟、通过区域固定和独立自动扩缩容控制成本的做法，代码已在 GitHub 仓库公开。

    Diagram depicting a person speaking to a voice bot composed of STT, LLM, and TTS services

    t and can return both partial and final transcription results. However, we found that in practice using Pipecat’s local VAD and turn detection to segment audio and passing that to NVIDIA’s parakeet-tdt-0.6b-v3 was faster than the open weights streaming implementations we tried or implementing our own VAD alongside the model. While this doesn’t provide the real-time feel of partial transcripts, when it comes down to total voice-to-voice latency, the only thing that matters is the final transcript

    匹配字段正文

    #company#教程/实践#语音#部署/工程

星期二

  1. vLLM 官方博客精选

    vLLM 上 Kimi K2 工具调用准确率的深度调试

    vLLM 与 Kimi 团队修复了 Kimi K2 的关键工具调用兼容问题，使成功解析的调用数从 218 个增至 971 个，提升 4.4x。问题涉及缺失 add_generation_prompt、空 content 的结构转换，以及历史工具调用 ID 不符合 functions.func_name:idx 格式。修复后仍出现 316 次 schema_validation_error_count，团队正协作将用于约束工具选择的 Enforcer 组件集成到 vLLM。

    helping me onboard the vLLM project and explain all the details of vLLM’s toolcall functionality. vLLM plays an important role in LLM serving, and diving deep into vLLM helps me understand the nuts and bolts of LLMs. Related Posts Day 0 Support for Qwen3.8-2.4T-A95B on vLLM Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.

    匹配字段正文

    #engineering#教程/实践#MCP/工具调用#部署/工程

星期五

  1. vLLM 官方博客

    深入 vLLM：高吞吐 LLM 推理系统架构解析

    文章系统解析 vLLM V1 如何从单 GPU 离线推理扩展到多 GPU、多节点异步服务。内容涵盖调度器、PagedAttention、连续批处理、前缀缓存、引导解码、推测解码、P/D 分离及分布式请求链路，并说明如何用 vllm bench 衡量延迟与吞吐量。分析基于 2025 年 8 月 9 日的 commit 42172ad。

    (GPU 0 for prefill and GPU 1 for decode), and then transfer the KV cache between them: Note: I've also experimented with LMCache [11], the fastest production-ready connector (uses NVIDIA's NIXL as the backend), but it's still at the bleeding edge and I ran into some bugs. Since much of its complexity lives in an external repo, SharedStorageConnector is a better choice for explanation. These are the steps in vLLM: Instantiation — During engine construction, connectors are created in two places: I

    匹配字段正文

    #engineering#教程/实践#部署/工程#开源生态

星期一

  1. vLLM 官方博客

    vLLM 使用 CUDA Core Dump 调试非法内存访问问题

    vLLM 介绍了如何启用 CUDA core dump，在非法内存访问发生时保存 GPU 状态，并用 cuda-gdb 定位出错的 CUDA kernel。文章给出推荐的环境变量组合，演示其定位普通 kernel 启动和 CUDA graph 内异常的流程，并说明生产环境性能影响、无法触发转储的错误类型及调试符号配置。

    system together. To share your story or usage with vLLM, please submit a PR at the blogpost repository. We would like to thank Ze Long, Vikram Sharma Mailthody, Jeremy Iverson, and Sandarbh Jain from NVIDIA for their helpful discussions. Lucas Wilkinson from Red Hat helped polishing the draft. Related Posts How vLLM developers debug hanging and complex CUDA kernels by triggering GPU core dumps, identifying stuck kernels, and mapping failures back to source code lines for faster kernel debugging.

    匹配字段正文

    #engineering#教程/实践#部署/工程

星期一

  1. vLLM 官方博客

    MiniMax-M1 混合架构如何借助 vLLM 实现长上下文快速推理

    vLLM 通过内存管理与底层计算优化支持 MiniMax-M1 的混合架构，实现长上下文高效推理与部署。MiniMax-M1 总参数量为 456B，推理时动态激活约 45.9B 参数；其 Lightning Attention 已通过 Triton 集成至 vLLM。文章还提供了使用 vLLM 和 Docker 部署该模型的快速指南。

    ng Mo and Kaichao You for their significant contributions. We also appreciate the efforts of the MiniMax engineering team, especially Gangying Qing, Jun Qing, and Jiaren Cai, whose dedication made this work possible. Related Posts Day 0 Support for Qwen3.8-2.4T-A95B on vLLM Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.

    匹配字段正文

    #engineering#教程/实践#部署/工程#推理

星期一

  1. Vapi Blog

    什么是 Gemma 3？Google 开源权重模型的能力与部署解析

    Vapi 博客介绍 Google 于 2025 年发布的开源权重模型 Gemma 3，它基于 Gemini 2.0 研究，支持文本、图像和视觉输入，可在单 GPU/TPU 上运行，支持 140+ 语言和最高 128k token 上下文，并提供 1B、4B、12B、27B 四种尺寸。文章称 Gemma 3 在人类偏好评测中优于 Llama3-405B 和 DeepSeek-V3，27B 版本在 LMArena Elo 榜得 1338 分，高于 DeepSeek-V3 的 1318 和 o3-mini 的 1304。经量化感知训练后，1B 模型显存占用最低 861 MB（INT4），27B 为 19.9 GB（INT4）；该模型已内置在 Vapi 控制台的 LLM 下拉菜单中。

    enMath, and MMMU) shows impressive efficiency gains. Gemma 3 27B scored 1338 on LMArena's Elo leaderboard, outperforming DeepSeek-V3 (1318) and o3-mini (1304) while using a single NVIDIA H100 GPU versus competitors requiring multiple accelerators. Processing speed proves crucial for voice applications. The 1B variant handles 2,585 tokens per second during prefill, creating sub-second response times that feel natural in conversation. This efficiency translates directly to cost savings and better

    匹配字段正文

    #company#产品更新#多模态#端侧

星期五

  1. vLLM 官方博客

    vLLM 如何集成 Transformers 建模后端

    vLLM 集成 Transformers 建模后端，让兼容模型在沿用 Transformers 实现的同时获得 vLLM 的吞吐量与延迟优化。该后端现已支持视觉语言模型，并可通过 model_impl="transformers"、OpenAI 兼容 API 或 LLM API 部署；对于 vLLM 尚未原生支持的兼容模型，系统也可自动切换到 Transformers 实现。

    lock a workflow that’s both flexible and scalable. Whether you’re prototyping a new model, deploying a custom creation, or scaling a multimodal app, this combination accelerates your path from research to production. Related Posts Day 0 Support for Qwen3.8-2.4T-A95B on vLLM Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.

    匹配字段正文

    #engineering#教程/实践#部署/工程#多模态

星期三

  1. vLLM 官方博客

    在 AMD MI300X 上用 vLLM 服务大语言模型的最佳实践

    vLLM 官方博客给出在 AMD MI300X 上部署大语言模型的优化指南，涵盖分块预填充、多步调度、前缀缓存、图捕获、KV cache 和张量并行等 8 项设置。测试中，vLLM 服务 Llama 3.1 405B 时比 TGI 吞吐量高 1.5x、TTFT 快 1.7x，服务 Llama 3.1 70B 时吞吐量高 1.8x、TTFT 快 5.1x。

    e AMD MAD repository, disabling it can prevent potential GPU hangs and improve overall efficiency. This can be achieved with the following command:; Tuning NCCL Communication: The NVIDIA Collective Communications Library (NCCL) is used for inter-GPU communication. For MI300X, the AMD vLLM fork performance document suggests setting the NCCL MIN NCHANNELS environment variable to 112 to potentially enhance performance In our tests, enabling these two configurations yielded a slight performance impr

    匹配字段正文

    #engineering#教程/实践#部署/工程#开源生态