Abstract : The main idea in test time training (TTT) ( 1) is that a model with fixed parameters produces the supervision for another network that is updated during test time (or inference time). This article first reviews the TTT paper. Then, we discuss the problem with TTT and how LaCT addresses them, resulting in ...
In DeltaProduct 1, they propose to improve DeltaNet 1 by updating the online memory with KVs for each token, which can be seen as performing multiple steps of gradient descent per token. I will explain how this method is almost the same as multi KV DeltaNet and reveal a potential flaw in the design of DeltaProduct.
This blog post is part 1 of a series that describes my attempt in implementing the Test Time Training (TTT) model proposed by 1, and Titans, proposed by 1. At the time of writing, these two are two strong recurrent language models, but they have not yet open sourced their implementation (TTT has only open sourced th...
1 | 1 TL;DR : A reader is augmented with a growing notebook that caches all edits in natural texts, and the reader retrieves relevant edits and make inference based on them. This achieves SOTA in model editing in QA and fact checking.
1 | 1 The first benchmark for evaluating the effectiveness of LLMs in handling more than 100k tokens! In the paper, we name it Bench, but I will sometimes use "InfiniteBench" in this blog post for better readability. Finally got some time to write this blog, been so busy lately! I have been in a fairly long duration...
1 I can't believe I haven't read this until now. This is mind provoking, and the result is an important step towards understanding neural networks.
1 TLDR: Propose ActAdd , a method for controlling model behavior during inference by modifying activations with a bias term that is learned from a pair of prompt. Summary: Propose ActAdd , a method for controlling model behavior by modifying activations at inference time. Steering vectors are computed by taking the ...
I will be holding a seminar at ModelBest (面壁智能) in Sep 20, 2023 in Beijing, Haidian, 科技园. The seminar will be in Chinese, and it's called "大模型安全与伦理问题" (translation: Safety and Ethical Concerns of Large Language Models). Below is a list of references.
1 | 1 | 1 | 1 I did this work with my girlfriend, whose research direction is computational fluid dynamics (CFD). We observed that there are numerous research works in applying deep learning (DL) to solve CFD problems. E.g., 1 have shown that DL methods can not only be more accurate than the best numerical methods, ...