搜索“GitHub”

into building good ones. The purpose of this cookbook is to enable other technical groups to hit the ground running when building their own hybrid (SSM, Transformer, MoE) models. GitHub For context, we at Zyphra have built the following hybrid models: BlackMamba; Zamba-7B; Zamba2-2.7B; Zamba2-1.2B The following datasets: Zyda And the following engineering optimizations Tree Attention Dense transformer models (i.e. alternating multi-head attention (MHA) and multilayer perceptron (MLP) blocks) ha

匹配字段正文

e of the richest coding data in the industry. Replit data is unique. We store over 300 million software repositories, in the same ballpark of the world’s largest coding hosts like GitHub and Bitbucket. We have a deeply granular understanding of the timeline of each project, thanks to a protocol called Operational Transformation (OT), and to the execution data logged when developers run their programs. The timeline is also enriched with error stack traces and LSP diagnostic logs, so we have debug

匹配字段正文

d the notion of a “human baseline” for harder tasks may cease to be meaningful. ‍ Future Outlook: We have released 80% of the LAB-Bench dataset today (available at HuggingFace and GitHub .) We are withholding a private test set of 20% that we will use to monitor for training contamination (models being overfit to the benchmark). Moreover, the process that we have developed internally for building these evals is scalable and extensible, and we expect to leverage it to create more challenging benc

匹配字段正文

e is manually labeled with fine-grained annotations on hallucination error types. Cited as: Weng, Lilian. (Jul 2024). Extrinsic Hallucinations in LLMs. Lil’Log. https://lilianweng.github.io/posts/2024-07-07-hallucination/. Or [1] Ji et al. “Survey of hallucination in natural language generation.” ACM Computing Surveys (2022) [2] Gekhman et al. “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?” arXiv preprint arXiv:2405.05904 (2024). [3] Min et al. “FActScore: Fine-grained atomic

匹配字段正文

documentation site. Assuming you have a chat model set up already (e.g. Codestral, Llama 3), you can keep this entire experience local by providing a link to the Ollama README on GitHub and asking questions to learn more with it as context. a. Type @docs in the chat sidebar, select “Add Docs”, copy and paste “https://github.com/ollama/ollama” into the URL field, and type “Ollama” into the title field b. It should quickly index the Ollama README and then you can ask it questions with important s

匹配字段正文

gh it is primarily a research codebase not intended for production use, its contributions to the field are significant. You can explore and contribute to the project here: https://github.com/yandex-research/swarm. SWARM is built upon the hivemind project: Hivemind Hivemind is a framework to perform various distributed training algorithms. It has been used to train a text to image model across a pool of volunteer compute. At its core, Hivemind utilizes a distributed hash table spread across each

匹配字段正文

orresponding short clip is synthesized sequentially conditioned on the keyframes Cited as: Weng, Lilian. (Apr 2024). Diffusion Models Video Generation. Lil’Log. https://lilianweng.github.io/posts/2024-04-12-diffusion-video/. Or [1] Cicek et al. 2016. “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation.” [2] Ho & Salimans, et al. “Video Diffusion Models.” 2022 | webpage [3] Bar-Tal et al. 2024 “Lumiere: A Space-Time Diffusion Model for Video Generation.” [4] Brooks et al. “Vi

匹配字段正文

understand what it's like to google "learn LLM"; then see 3 sponsored posts asking you to download e-books (in which nothing concrete appears)... then see 20 ultimate roadmaps and GitHub repos, where step 1/54 is to view a 2-hour long video (and no one knows what step 54 is because it's so looooooooong). So, instead of putting you through this pain myself, let's just break down what LLMs are in 3 key ideas: In ChatGPT, you input a piece of text, and it returns text, right? Well, what's actually

匹配字段正文

s a popular open-source framework by Microsoft for building multi-agent applications. For this, example we’ll use the Code Llama model: Install Autogen: Then create a Python script example.py to use Ollama with Autogen: Lastly, run the example to have the assistant write the code to plot a chart: More to come This is initial experimental support for the OpenAI API. Future improvements under consideration include: GitHub issues are welcome! For more information, see the OpenAI compatibility docs.

匹配字段正文

e trusted candidate set $\mathcal{C}$ and more noisy samples are removed. Cited as: Weng, Lilian. (Feb 2024). “Thinking about High-Quality Human Data”. Lil’Log. https://lilianweng.github.io/posts/2024-02-05-human-data-quality/. Or [1] Francis Galton “Vox populi” Nature 75, 450-451 (1907). [2] Sambasivan et al. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI" CHI 2021 [3] Chris Callison-Burch. “Fast, Cheap, and Creative: Evaluating Translation Quality Usi

匹配字段正文

team at Allen Institute for AI , whose API for exploring the citation graph of science is a key feature of PaperQA. We plan to make the full WikiCrow and PaperQA code available on GitHub soon. Until then, the essential aspects of the PaperQA algorithm are available (although you will need access to your own repository of full text scientific articles), as well as the prompts used for WikiCrow . Benchmarking PaperQA In our evaluations, PaperQA outperforms GPT-4, Perplexity, and other LLMs, as wel

匹配字段正文

ctiveness is measured by attack objective functions designed for different experiments: - In text-to-image experiment, they used Q16 (Schramowski et al. 2022) and NudeNet (https://github.com/notAI-tech/NudeNet). - text-to-text experiment: TOXIGEN; Diversity is measured by pairwise dissimilarity, in form of $\sum {(\mathbf{x} i, \mathbf{x} j) \in \text{All pairs}} [1 - \text{sim}(\mathbf{x} i, \mathbf{x} j)]$; Low-toxicity is measured by Perspective API; Scoring-LIFO: Combine LIFO and Scoring str

匹配字段正文

任务高度相关的目标算法，并且处理器是使用基于这些算法构建的定制执行数据集来训练的。这通常既需要领域专业知识，也需要计算机科学专业知识，因此构成了明显的入门门槛！ 自从我开始从事NAR以来，我一直对降低这一入门门槛抱有浓厚兴趣，同时也希望提供一组在各种任务中都应该有用的“基础处理器”。这促成了为期两年的工程努力，最终开源了CLRS算法推理基准（https://github.com/deepmind/clrs）[44]。 CLRS基准的灵感来自Cormen、Leiserson、Rivest和Stein的经典教材《算法导论》（CLRS）[1]，这是关于经典算法和数据结构最受欢迎的本科教材之一，也是竞赛程序员的“圣经”。尽管它有数千页之多，却只包含约90种不同的算法，而这些算法往往构成了整个软件工程职业生涯的基础！因此，CLRS中的算法可以构成我们所需的坚实的“基础处理器”集合，我们着手让构建和训练NAR处理器来执行CLRS中的算法变得容易。 在其第一个版本中，我们发布了CLRS-30，这是CLRS中三十种算法的数据集和处理器生成器集合，涵盖多种技能：排序、搜索、动态规划、图算法、字符串算法和

匹配字段正文

mechanisms; Try it: ollama run nous-hermes-llama2; Eric Hartford’s Wizard Vicuna 13B uncensored Llama 1 13B model fine-tuned to remove alignment; Try it: ollama run wizard-vicuna Example output comparisons We’ve tried running the 7B Llama 2 model against the 7B llama2 uncensored model with the same prompts. Below are the outputs. Disclaimer: Uncensored models carry their own risk. Please use them accordingly. Try running uncensored models yourself with Ollama: https://github.com/jmorganca/ollama

匹配字段正文

uction). Consequently, much of the agent demo code focuses on parsing model output Cited as: Weng, Lilian. (Jun 2023). “LLM-powered Autonomous Agents”. Lil’Log. https://lilianweng.github.io/posts/2023-06-23-agent/. Or [1] Wei et al. “Chain of thought prompting elicits reasoning in large language models.” NeurIPS 2022 [2] Yao et al. “Tree of Thoughts: Dliberate Problem Solving with Large Language Models.” arXiv preprint arXiv:2305.10601 (2023). [3] Liu et al. “Chain of Hindsight Aligns Language M

匹配字段正文

GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM

匹配字段正文

ging with LLMs at the core. While there isn’t a lot of formal education available on this topic yet, we pulled out some of the most useful resources we’ve found. Reference Build a GitHub support bot with GPT3, LangChain, and Python: One of the earliest public explanations of the modern LLM app stack. Some of the advice in here is dated, but in many ways it kicked off widespread adoption and experimentation of new AI apps; Building LLM applications for production: Chip Huyen discusses many of the

匹配字段正文

onfidence measures to achieve good performance-efficiency tradeoffs Cited as: Weng, Lilian. (Jan 2023). Large Transformer Model Inference Optimization. Lil’Log. https://lilianweng.github.io/posts/2023-01-10-inference-optimization/. Or [1] Bondarenko et al. “Understanding and overcoming the challenges of efficient transformer quantization” ACL 2021. [2] Dettmers et al. “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale” NeuriPS 2022 [3] Zadeh et al. “Gobo: Quantizing attention-bas

匹配字段正文

mmend this post for more discussion on linearized models and lazy training. Cited as: Weng, Lilian. (Sep 2022). Some math behind neural tangent kernel. Lil’Log. https://lilianweng.github.io/posts/2022-09-08-ntk/. Or [1] Jacot et al. “Neural Tangent Kernel: Convergence and Generalization in Neural Networks.” NeuriPS 2018. [2]Radford M. Neal. “Priors for Infinite Networks.” Bayesian Learning for Neural Networks. Springer, New York, NY, 1996. 29-53. [3] Lee & Bahri et al. “Deep Neural Networks as G

匹配字段正文

. Please sign up here if you'd like to help us test it. Here is what alpha users are saying about it: "The first thing me and all my friends noticed was how much faster it is than GitHub Copilot. It is at least 2x faster, maybe 3x. It's a little detail but it makes a big difference."; "It makes web development so much easier. I feel like I'm only writing 50% of the code."; "After using the feature for only week, I can't imagine life without it"; "It's crazy how much faster I can learn new things

匹配字段正文