titans-unofficial

About

Titans pairs attention, which acts as short-term memory, with a small neural network that serves as long-term memory. That network keeps learning while the model reads: it is updated by gradient descent on each token, at inference time as well as during training. Includes the paper's architecture variants.