←搜索结果

搜索“Agent”

按匹配度 · 共 404 条结果 清除搜索

星期一

  1. vLLM 官方博客

    vLLM 分布式推理中的张量并行与流水线并行

    vLLM 介绍了如何结合张量并行和流水线并行，在多 GPU 或多节点上部署超出单卡容量的大模型。张量并行可降低延迟但依赖高带宽互连，流水线并行通信开销较低且更适合跨节点部署。案例中从 TP=1 扩展到 TP=2 后，KV cache block 增加 13.9x，token 吞吐量提升 3.9x。

    ptimizing AI workloads. Come to the Bi-weekly Office Hours to learn more about LLM inference optimizations and vLLM! Acknowledgement Sangbin Cho (xAI) for the origination of some of the figures. Related Posts vLLM x AgentX: Optimizing for Real-World Agentic Serving How vLLM optimizes KV cache management, parallelism, scheduling, and P/D disaggregation for agentic workloads, validated on SemiAnalysis AgentX with up to 130K tokens per GPU-second and a 14.6x-106x serving-cost advantage over Opus 5.

    匹配字段正文

    #engineering#教程/实践#部署/工程#推理

星期二

  1. vLLM 官方博客

    vLLM 结构化解码入门与 XGrammar 集成解析

    vLLM 介绍结构化解码的原理与实现，并已支持 Outlines 和 XGrammar 两种后端，以约束大语言模型输出符合指定结构。XGrammar 集成在负载下可将每个输出 token 的生成时间改善至多 5x，不适用的请求会回退至 Outlines。规划中的 v1 将重点优化性能，并探索调度器级掩码广播以支持混合请求批处理。

    top level view of structure decoding

    , and in the process trying to posit that structured decoding represents a fundamental shift in how we think about LLM outputs. It also plays an important role in building complex agentic system. For more information about vLLM, please check out our documentation. Language models: A brief historical context In 1950, Alan Turing proposed that a high-speed digital computer, programmed with rules, could exhibit emergent behaviour of intelligence (Turing, 1950). This led to two main approaches in AI

    匹配字段正文

    #engineering#教程/实践#部署/工程#Agent

星期三

  1. vLLM 官方博客

    在 AMD MI300X 上用 vLLM 服务大语言模型的最佳实践

    vLLM 官方博客给出在 AMD MI300X 上部署大语言模型的优化指南，涵盖分块预填充、多步调度、前缀缓存、图捕获、KV cache 和张量并行等 8 项设置。测试中，vLLM 服务 Llama 3.1 405B 时比 TGI 吞吐量高 1.5x、TTFT 快 1.7x，服务 Llama 3.1 70B 时吞吐量高 1.8x、TTFT 快 5.1x。

    it by docker pull ghcr.io/embeddedllm/vllm-rocm:cb3b2b9). All of the benchmarks are run in the docker container instance, and are run with 4 MI300X GPUs using CK Flash Attention with VLLM USE TRITON FLASH ATTN=0. Detail Benchmark Configuration Related Posts Announcing vllm-metal: Concurrent Serving on Apple Silicon vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M5 prefill acceleration.

    匹配字段正文

    #engineering#教程/实践#部署/工程#开源生态

星期五

  1. Lilian Weng：Lil'Log

    LLM 驱动的自主智能体：规划、记忆与工具调用综述

    Lilian Weng 在 Lil'Log 撰文梳理以 LLM 为核心控制器的自主智能体架构，将其拆解为规划、记忆和工具调用三大组件。规划部分涵盖 CoT、Tree of Thoughts、LLM+P 等任务分解方法，以及 ReAct、Reflexion、Chain of Hindsight、Algorithm Distillation 等自我反思机制。记忆部分区分短期上下文学习与基于外部向量库的长期记忆，并介绍 LSH、ANNOY、HNSW、FAISS、ScaNN 等快速 MIPS 检索算法；工具调用部分则列举 MRKL、TALM、Toolformer、HuggingGPT、API-Bank 以及 ChemCrow 等科学发现智能体案例。文末指出有限上下文长度、长期规划与任务分解、自然语言接口可靠性三类常见局限。

    Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver. In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components: Planning Subgoal and decomposit

    匹配字段正文

    #analysis#教程/实践#Agent#推理