搜索“GitHub”

mmend this post for more discussion on linearized models and lazy training. Cited as: Weng, Lilian. (Sep 2022). Some math behind neural tangent kernel. Lil’Log. https://lilianweng.github.io/posts/2022-09-08-ntk/. Or [1] Jacot et al. “Neural Tangent Kernel: Convergence and Generalization in Neural Networks.” NeuriPS 2018. [2]Radford M. Neal. “Priors for Infinite Networks.” Bayesian Learning for Neural Networks. Springer, New York, NY, 1996. 29-53. [3] Lee & Bahri et al. “Deep Neural Networks as G

匹配字段正文

. Please sign up here if you'd like to help us test it. Here is what alpha users are saying about it: "The first thing me and all my friends noticed was how much faster it is than GitHub Copilot. It is at least 2x faster, maybe 3x. It's a little detail but it makes a big difference."; "It makes web development so much easier. I feel like I'm only writing 50% of the code."; "After using the feature for only week, I can't imagine life without it"; "It's crazy how much faster I can learn new things

匹配字段正文

rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips Cited as: Weng, Lilian. (Jun 2022). Generalized visual language models. Lil’Log. https://lilianweng.github.io/posts/2022-06-09-vlm/. Or [1] Li et al. “VisualBERT: A Simple and Performant Baseline for Vision and Language.” arXiv preprint:1908.03557 (2019). [2] Wang et al. “SimVLM: Simple Visual Language Model Pretraining with Weak Supervision.” ICLR 2022. [3] Aghajanyan, et al. “CM3: A Causal Masked Multimodal Model o

匹配字段正文

e high or the corruption transition matrix is not symmetric. Cited as: Weng, Lilian. (Apr 2022). Learning with not enough data part 3: data generation. Lil’Log. https://lilianweng.github.io/posts/2022-04-15-data-gen/. Or [1] Zhang et al. “Adversarial AutoAgument” ICLR 2020. [2] Kumar et al. “Data Augmentation using Pre-trained Transformer Models.” AACL 2020 Workshop. [3] Anaby-Tavor et al. “Not enough data? Deep learning to rescue!” AAAI 2020. [4] Wang et al. “Want To Reduce Labeling Cost? GPT-3

匹配字段正文

etter in time, the threshold $\delta$ decays in time as well Cited as: Weng, Lilian. (Feb 2022). Learning with not enough data part 2: active learning. Lil’Log. https://lilianweng.github.io/posts/2022-02-20-active-learning/. Or [1] Burr Settles. Active learning literature survey. University of Wisconsin, Madison, 52(55-66):11, 2010. [3] Yang et al. “Cost-effective active learning for deep image classification” TCSVT 2016. [4] Yarin Gal et al. “Dropout as a Bayesian Approximation: representing mo

匹配字段正文

eudo label distribution to reduce the class overlap Cited as: Weng, Lilian. (Dec 2021). Learning with not enough data part 1: semi-supervised learning. Lil’Log. https://lilianweng.github.io/posts/2021-12-05-semi-supervised/. Or [1] Ouali, Hudelot & Tami. “An Overview of Deep Semi-Supervised Learning” arXiv preprint arXiv:2006.05278 (2020). [2] Sajjadi, Javanmardi & Tasdizen “Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning.” arXiv preprint arXiv:

匹配字段正文

utation, constant buffer size and on-the-fly memory defragmentation. Cited as: Weng, Lilian. (Sep 2021). How to train really large models on many GPUs? Lil’Log. https://lilianweng.github.io/posts/2021-09-25-train-large/. Or [1] Li et al. “PyTorch Distributed: Experiences on Accelerating Data Parallel Training” VLDB 2020. [2] Cui et al. “GeePS: Scalable deep learning on distributed GPUs with a GPU-specialized parameter server” EuroSys 2016 [3] Shoeybi et al. “Megatron-LM: Training Multi-Billion P

匹配字段正文

I datasets, IS-BERT produces results comparable with SBERT (See Fig. 25 & 30). Cited as: Weng, Lilian. (May 2021). Contrastive representation learning. Lil’Log. https://lilianweng.github.io/posts/2021-05-31-contrastive/. Or [1] Sumit Chopra, Raia Hadsell and Yann LeCun. “Learning a similarity metric discriminatively, with application to face verification.” CVPR 2005. [2] Florian Schroff, Dmitry Kalenichenko and James Philbin. “FaceNet: A Unified Embedding for Face Recognition and Clustering.” CV

匹配字段正文

studying the risk of neural toxic degeneration in language models. [Download] Cited as: Weng, Lilian. (Mar 2021). Reducing toxicity in language models. Lil’Log. https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/. Or [1] Vidgen, et al. “Challenges and frontiers in abusive content detection.” Workshop on Abusive Language Online 2019. [2] Zampieri et al. “Predicting the type and target of offensive posts in social media.” NAACL 2019. [3] Vidgen & Deczynski. “Directions in abusive language t

匹配字段正文

and masked. The task is to predict this masked salient span Cited as: Weng, Lilian. (Oct 2020). How to build an open-domain question answering system? Lil’Log. https://lilianweng.github.io/posts/2020-10-29-odqa/. Or SQuAD 2.0: the Stanford QA dataset; RACE: a reading comprehension dataset collected from English Examinations that are created for middle school and high school students; TREC QA: the TREC QA collections; MS MARCO: a QA dataset featuring 100,000 real Bing questions and a human gener

匹配字段正文

th a random choice (e.g. “swap the output address” or “change the value of a constant”) Cited as: Weng, Lilian. (Aug 2020). Neural architecture search. Lil’Log. https://lilianweng.github.io/posts/2020-08-06-nas/. Or [1] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter. “Neural Architecture Search: A Survey” JMLR 20 (2019) 1-21. [2] Kenneth O. Stanley, et al. “Designing neural networks through neuroevolution” Nature Machine Intelligence volume 1, pages 24–35 (2019). [3] Kenneth O. Stanley & Risto

匹配字段正文

atent skill variable. See my previous post for more details. Cited as: Weng, Lilian. (Jun 2020). Exploration strategies in deep reinforcement learning. Lil’Log. https://lilianweng.github.io/posts/2020-06-07-exploration-drl/. Or [1] Pierre-Yves Oudeyer & Frederic Kaplan. “How can we define intrinsic motivation?” Conf. on Epigenetic Robotics, 2008. [2] Marc G. Bellemare, et al. “Unifying Count-Based Exploration and Intrinsic Motivation”. NIPS 2016. [3] Georg Ostrovski, et al. “Count-Based Explorat

匹配字段正文

xt{RL}(\theta \vert \pi \text{mm}) + \lambda \mathcal{L} \text{mm}(\theta)$. Cited as: Weng, Lilian. (Jan 2020). Curriculum for reinforcement learning. Lil’Log. https://lilianweng.github.io/posts/2020-01-29-curriculum-rl/. Or [1] Jeffrey L. Elman. “Learning and development in neural networks: The importance of starting small.” Cognition 48.1 (1993): 71-99. [2] Yoshua Bengio, et al. “Curriculum learning.” ICML 2009. [3] Daphna Weinshall, Gad Cohen, and Dan Amir. “Curriculum learning by transfer l

匹配字段正文

l the interesting findings above, it should be pretty fun to reproduce them. Some results are easily to reproduce than others. Details are described below. My code is available on github lilianweng/generalization-experiment. This is the trickiest one to reproduce. The authors did give me a lot of good advice and I appreciate it a lot. Here are a couple of noticeable settings in their experiments: There are no regularization terms like weight decay, dropout; In Fig 3, the training set contains 4k

匹配字段正文

same type may vary a lot across multiple robots. Batch normalization is applied to fix it by normalizing every dimension across samples in one minibatch. D4PG [paper|code (Search “github d4pg” and you will see a few.)] Distributed Distributional DDPG (D4PG) applies a set of improvements on DDPG to make it run in the distributional fashion. (1) Distributional Critic: The critic estimates the expected Q value as a random variable a distribution $Z w$ parameterized by $w$ and therefore $Q w(s, a) =

匹配字段正文

folder. TensorBoard will read this file during startup. The model is trained with top 50 stocks with largest market values in the S&P 500 index. (Run the following command within github.com/lilianweng/stock-rnn) And the following configuration is used: Price Prediction As a brief overview of the prediction quality, Fig. 3 plots the predictions for test data of “KO”, “AAPL”, “GOOG” and “NFLX”. The overall trends matched up between the true values and the predictions. Considering how the predicti

匹配字段正文

open source toolkits and libraries for building deep learning models. Tensorflow is fairly new but has attracted a lot of popularity. It turns out, TensorFlow was the most forked Github project of 2015. All that happened in a period of 2 months after its release in Nov 2015. If you are very new to the field and willing to devote some time to studying deep learning in a more systematic way, I would recommend you to start with the book Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Cou

匹配字段正文