←搜索结果

搜索“GitHub”

按匹配度 · 共 369 条结果 清除搜索

星期一

  1. Prime Intellect

    Prime Intellect 开源分布式推理栈，面向消费级 GPU 与公网高延迟

    Prime Intellect 发布分布式推理栈预览，并开源 PRIME-IROH、PRIME-VLLM、PRIME-PIPELINE 三个代码库，目标是让消费级 GPU 通过公网连接运行推理。文章指出流水线并行通信需求最低，适合分布式环境，但同步流水线会因 100ms 级跨洲延迟形成吞吐上限，而异步微批调度在内存带宽受限的解码阶段无法带来训练那样的加速。团队称已可用私有硬件通过公网跑推理，正把组件接入自家栈，用于 SYNTHETIC-2 合成数据运行。

    k FLOPs per byte that we can expect from our hardware. For example, for a H100 the accelerator intensity is 989e12/3.35e12≈295989e^{12} / 3.35e^{12} \approx 295989e12/3.35e12≈295 (GitHub Gist). By simple re-arrangement we get that we are in the compute-bound regime for operation intensities that exceed our accelerator intensity. Example: Dot Product Let’s look at a simple example: Consider the dot product (∘\circ∘) between two nnn-dimensional vectors u,v∈Rnu, v \in \mathbb{R}^nu,v∈Rn defined as

    匹配字段正文

    #company#开源/仓库#部署/工程#推理

星期五

  1. antirez 原创博客

    Redis 合并 Vector Sets，成为 Redis 8 原生数据类型

    Redis 作者 antirez 宣布 Vector Sets 已合并进 Redis，从 Redis 8 起作为原生数据类型提供，目前仍是 beta 功能。它采用类似 Sorted Set 的接口，用 VADD 添加、VREM 删除元素，VSIM 做向量相似查询，并支持 8 bit 与二进制量化，其中 int8 量化占用 FP32 约 25% 内存。作者重写了 HNSW 实现以支持真正的节点删除和带谓词的图扫描，VSIM 请求走多线程，Vector Sets 以图的形式序列化到磁盘，重启后加载每百万元素只需几秒。

    also reimplement from scratch HNSWs (you can see my implementation in hnsw.c), because that was going to be my core data structure, and I didn’t want to grab some random code from GitHub and be happy with it. However, as I started reading the papers, I started to understand that a few pieces were missing. So, as I did in the past with HyperLogLog, where I had to fill a few gaps (here: https://antirez.com/news/75 ), there was already some new algorithmic challenges. Especially I wanted two things

    匹配字段正文

    #analysis#产品更新#数据/训练#开源生态

星期一

  1. Transluce

    Transluce 发布 Docent 技术预览，用于分析 AI 智能体轨迹

    Transluce 发布 Docent 的公开技术预览，这是一个用语言模型对智能体轨迹做摘要、聚类、搜索并支持反事实干预的系统，用户可在预置基准上在线查看轨迹。文中称在 InterCode 基准上仅补上缺失的 apt 与 pip 包，GPT-4o 的解决率就从 68.6% 提升到 78%；Docent 还发现 GPT-4o 在反复失败后输出“chestnut facts”等无关内容，以及 Cybench 任务中模型通过读取 Dockerfile 走捷径解题。Transluce 正在招募早期用户，并计划在数周内正式发布。

    Agent needs internet access to solve the task.

    iv preprint arXiv:2502.03461. [link]; Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023). Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. [link]; Chowdhury, N., Aung, J., Shern, C. J., Jaffe, O., Sherburn, D., Starace, G., Mays, E., Dias, R., Aljubeh, M., Glaese, M., Jimenez, C. E., Yang, J., Liu, K., & Madry, A. (2024). Introducing swe-bench verified. OpenAI. [link]; Dunlap, L., Mandal, K., Darrell, T., St

    匹配字段正文

    #company#产品更新#Agent#部署/工程

星期二

  1. vLLM 官方博客精选

    vLLM 发布面向 Kubernetes 的开源推理栈 production-stack

    vLLM 与 LMCache 团队发布开源参考实现 vLLM production-stack，将单节点 vLLM 扩展为可在 GPU 集群运行的完整推理服务。它支持 KV cache 共享与存储、前缀感知路由、TTFT 和 TBT 等指标观测及自动扩缩容，并可通过 Helm chart 部署到 Kubernetes。多轮问答基准结果显示，该方案在首个 token 延迟和 token 间延迟方面优于 vLLM + KServe 与一个商业端点服务。

    ons seeking to build, test, and deploy LLM applications at scale without sacrificing performance or simplicity. If you’re as excited as we are, don’t wait! Clone the repo: https://github.com/vllm-project/production-stack; Kick the tires; Let us know what you think!; Interest Form Join us to build a future where every application can harness the power of LLM inference—reliably, at scale, and without breaking a sweat. Happy deploying! What AIBrix adds as a Kubernetes control plane for vLLM: LoRA m

    匹配字段正文

    #engineering#产品更新#部署/工程#开源生态

星期一

  1. Cognition 模型 / Devin 博客精选

    Devin 正式开放订阅，起价每月 500 美元

    Cognition 宣布 Devin 正式开放，所有工程团队均可使用它修复前端 bug、为待办任务生成初稿 PR、执行重构等，订阅起价每月 500 美元，包含不限席位、API、Slack 集成和 IDE 扩展。过去两周 Devin 提速约 10%、成本效率提升约 10%，同类任务消耗的 Agent Compute Units 更少。本次更新还修复了崩溃、卡死和挂起问题，并新增会话过滤、Slack 通知、机器规格调整和可固定的仓库 Knowledge 等配置项。

    her Devin sessions start in existing or new Slack threads, whether Devin waits for you to approve its plan, and more in Settings Customization. Devin can send Slack updates on its Github activity. Configure the channel these updates are sent to in Settings Integrations. Share Devin sessions you start in the web app to Slack. You can now change the default channel. Configure + monitor Devin’s machine If you need to increase Devin’s machine size (disk space, RAM, CPU), we’ve added additional optio

    匹配字段正文

    #company#产品更新#Agent#部署/工程

星期四

  1. vLLM 官方博客

    vLLM v0.6.0 发布，吞吐量提升最高 2.7x、TPOT 最多降低 5x

    vLLM 发布 v0.6.0，相比 v0.5.3 吞吐量提升 1.8-2.7x，其中 Llama 8B 的吞吐量提升 2.7x、TPOT 加快 5x，Llama 70B 的吞吐量提升 1.8x、TPOT 减少 2x。主要优化包括拆分 API server 与推理引擎、多步批量调度、异步输出处理、对象缓存和非阻塞数据传输。多步调度目前会造成 token 间延迟波动，并可能在低请求率下提高 TTFT。

    as well as captured detailed metrics for time-to-first-token (TTFT) and time-per-output-token (TPOT), please refer to the appendix for more data and analysis. You can follow this github issue to reproduce our benchmark. Limitation of current optimizations. Although our current optimizations give a significant throughput gain, there are performance trade-offs from our current optimizations, especially from multi-step scheduling: Bumpy inter-token latency: In our current implementation of multi-s

    匹配字段正文

    #engineering#产品更新#部署/工程#开源生态

星期四

  1. Ollama Blog

    Ollama 内置兼容 OpenAI Chat Completions API

    Ollama 宣布内置兼容 OpenAI Chat Completions API，可在本地使用更多工具和应用。使用时只需把请求的 hostname 改为 http://localhost:11434，即可沿用 OpenAI 的调用格式，官方还给出了 cURL 和 Microsoft 开源多智能体框架 Autogen 的接入示例。该支持目前为实验性，后续改进方向仍在考虑中。

    s a popular open-source framework by Microsoft for building multi-agent applications. For this, example we’ll use the Code Llama model: Install Autogen: Then create a Python script example.py to use Ollama with Autogen: Lastly, run the example to have the assistant write the code to plot a chart: More to come This is initial experimental support for the OpenAI API. Future improvements under consideration include: GitHub issues are welcome! For more information, see the OpenAI compatibility docs.

    匹配字段正文

    #company#产品更新#OpenAI#部署/工程

星期二

  1. vLLM 官方博客精选

    vLLM 发布基于 PagedAttention 的开源 LLM 推理与服务库

    vLLM 发布开源 LLM 推理与服务库，利用 PagedAttention 管理 KV cache，无需修改模型架构即可实现比 HuggingFace Transformers 最高 24x 的吞吐量。实验中其吞吐量比 TGI 最高提升 3.5x，内存浪费降至 4% 以下；LMSYS 部署后将服务所需 GPU 数量减少 50%。

    GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM

    匹配字段正文

    #engineering#产品更新#部署/工程#开源生态

星期四

  1. Replit Blog精选

    Replit 发布 Ghostwriter AI 编程助手，Complete Code 进入封闭测试

    Replit 宣布推出 Ghostwriter，把 AI 能力接入几乎全部 IDE 功能，旗舰功能 Complete Code 是实时补全代码的 AI 结对程序员，目前处于封闭测试。官方称通过 FasterTransformer 与 Triton 推理、知识蒸馏和量化等优化，中位响应时间低于 400ms；Explain Code、Generate Code、Transform Code 则调用 OpenAI 等更大模型。Ghostwriter 后续将打包为基于 Cycles 的付费功能，最终计划像 Multiplayer 一样默认免费开放。

    Ghostwriter AI & Complete Code Beta — cover image

    . Please sign up here if you'd like to help us test it. Here is what alpha users are saying about it: "The first thing me and all my friends noticed was how much faster it is than GitHub Copilot. It is at least 2x faster, maybe 3x. It's a little detail but it makes a big difference."; "It makes web development so much easier. I feel like I'm only writing 50% of the code."; "After using the feature for only week, I can't imagine life without it"; "It's crazy how much faster I can learn new things

    匹配字段正文

    #company#产品更新#编码#部署/工程