All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer questions about the text coherently.

But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases. They must start to generalize. A human could not read Infinite Jest in one pass and then answer details about every single sentence. But could a transformer, or a RNN? As the context grows, is it harder to keep a high granularity of detail? Or am I wrong in trying to think of these models the way I think about the human mind, and they are actually able to handle this problem just fine?

I'm aware that we can cheat a bit, by adding a lookup step into an embedding database, to provide "infinite" context with "infinite" precision. But to me, that is analogous to a human looking up information in a library in order to answer a question. I'm interested in the inherent, emergent memory that these models have available to them in just one forward pass.

What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding.

You can view it as every token embedding mixing information from other embeddings into itself. Done ~100 times in parallel ("attention heads"), ~100 times in sequence (layers). As per GPT-3 model (175B).

Details aside, math suggests they _must_ be using some sparse attention method: the memory used by attention is O(l^2 * d) (here O notation is a bit of overkill: it's exactly that number in 8 bit quantization or twice that in 16 bit floats). With l=32k, and d probably in the order of a few k (even "old" BERT model had it at 768, it only went up since), that would be of the order of a few TB. For (a piece of) _a single_ layer, they probably have dozens, if not hundreds, of it. The largest GPUs in commerce have 80GB of memory; there's no way they are really using that many GPUs for each single layer (even leaving aside the fact that runtime would probably be absolutely horrible).

Implicitly or explicitly, the transformer must learn to have both a "summarized" version of attention (that tells it where to focus on) and more "detailed" ones. It doesn't need to be specifically coded, if using sparse attention, the model might for example get different levels of attentions at different layers, somehow, but they probably have coded something a bit smarter than that.

[1] https://kir-gadjello.github.io/posts/gpt4-some-technical-hyp... [2] https://arxiv.org/abs/2111.12763

> it takes a fraction of every other token embedding and adds it to itself.

> every token embedding mixing information from other embeddings into itself

Noting the use of the word every. Phrased this way, calling it "attention" hardly makes sense, as attention is typically focused on something specific at any given time - not always on the entire thing.

The mixing step, is computed with the trained weights, meaning the model does learn on its own, when/what to emphasise in this "mixing" process.

Hypothetically speaking, if a token does not matter, it mixing weights could end up being literally 0 (or something close to it)

This could be seen as "soft attention" where in theory there would be few winners for every "attention head".

It's also possible that the only purpose the softmax actually serves (or at least major component of) is that of normalization. Without it, the variance in the internal network dynamics between training samples would be fairly large (some training samples may have tokens that interact heavily, while others not). Making optimization problematic.

And, in the case of BERT, every token embedding after it too.

But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative

Hallucination used to mean the following. A basic neural network is:

f(x) = y = repeat(nonlinearity(ax[0] + bx[1] + ...))

And then you adjust a, b, c, ... until y is reasonable, according to the cost function. But look! The very same backpropagation can adjust x[0], x[1] ... with the same cost function and only a small change in the code.

This allows you to reverse the question neural networks answer. Which can be an incredibly powerful way to answer questions.

And that used to be called hallucination in Neural networks. Instead of "change these network weights to transform x into y, keeping x constant" you ask "change x to transform x into y, keeping the network weights constant".

Now it's impossible finding half the papers on the this topic. AARGH!

It's most famous as a problem with ChatGPT specifically, which is presented as a chat interface where an agent answers your question. In that context, it makes sense to think of confident and detailed wrong answers as hallucinations.

You could say that it's still an LLM underneath, but then you'd be talking about a layer of abstraction beneath what most people interact with. Given that they tap into the mental model of "chat with an agent" heavily with their interface and interactions, having small disclaimers and saying "I'm just an LLM" from time to time aren't sufficient to counter people's intuitions and expectations.

Every-time someone comes up and say "this is not attention, because it does X and not Y"

My response is, ok, what would you call it then? Because no one (including me) seem to to able to find a better term, that fits its use case.

War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens long.

Many people could reason about the book in its entirety. They would not have an encyclopedic recall of random dates and side characters or be able to quote any passage, but they could perform deep analysis on it.

I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps with some loss of fidelity.

My guess is that there will need to be an advancement or two before we can can get an AI to read all of the ASIOF books so far and ask "What really happened at the Tower of Joy?"

Some of them, some of the time. This is best comparable with ChatGPT having those books in its training dataset.

The context window is more like short-term memory. GPT-4 can fit[0] ~1.5 chapters of Game of Thrones; GPT-4-32k almost six. Making space for prompt, questions and replies, say one chapter for GPT-4, and five chapters for GPT-4-32k.

Can you imagine having a whole chapter in your working memory at once? Being simultaneously aware of every word, every space, every comma, every turn of phrase, every character and every plot line mentioned in it - and then being able to take it all into account when answering questions? Humans can do it for a paragraph, a stanza, maybe half a page. Not a whole chapter in a novel. Definitely not five. Not simultaneously at every level.

I feel in this sense, LLMs already surpassed our low-level capacity - though the comparison is a bit flawed, since our short-term memory also keeps tracks of sights, sounds, smells, time, etc. and emotions. My point here isn't really to compare who has more space for short-term recall - it's to point out that answering questions about immediately read text is another narrow, focused task which machines can now do better than us.

----

[0] - 298000 words in the book (via [1]), over 72 chapters (via [2]), gives us 4139 words per chapter. Multiplying by 4/3, we get 5519 tokens per chapter. GPT-4-8k can fit 1.45x that; GPT-4-32k can fit 5.8x that.

[1] - https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula...

[2] - https://awoiaf.westeros.org/index.php/Chapters_Table_of_cont...

We take out key points explicitly so it's not summarized, and for the rest (less important parts) we summarize it and save it.

That would very likely fit and it would probably yield equal to or better recall and understanding than humans.

> At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret.

Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know.

There's a bunch of prior art for adaption techniques for getting new data into a trained model (fine tuning, RLHF etc). There's no real reason to think there won't be more techniques that turn what think of now as the context window into something that alters the weights in the model and is serialized back to disk.

But also, not just ASIOF is in the training set, but presumably lots of discussion about it and all the interesting events in the book.

That in essence, I believe, is the difference. An LLM (while still predicting the next word, given it's context), seem to care less about the number of subjects in a given context, than humans do.

[1]: https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus...

That’s analogous to having War and Peace in the training set. When actually reading War and Peace, nobody‘s working memory includes a precise recall of everything they’ve read thus far, which is more analogous to an LLM’s context size.

Summarizing chapters with the very same LLM and including that as context may very well get you far.

These things are very different. The mental model I have is that they put the entire book into a big hypercube and then *flash*, instantly, out comes the next token. And likewise, when they encounter text that is 8001 tokens long, token #1 is completely erased, gone, like it never existed.

You're confusing your perception of how your mind works with how your mind actually works.

Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no circumstances is human intuition about the human mind a reliable indicator for what is actually going on.

This is like asking ChatGPT why it came up with a specific response. It will give you an answer, of course, but that answer is generated the same way all its answers are – ChatGPT doesn't suddenly turn on a magic "introspection mode" that allows it to examine its internals. And there's no reason to assume that humans have such an introspection mode either. In fact, many results from experimental psychology seem to indicate that humans don't understand their own mental operations at all.

What we cannot do is make definitive claims about neurological processes and structures based on what we know about our cognitive processes.

Is there any actual evidence that our so-called reflections on our own thinking are anything more than hallucinations?

"It's obvious" doesn't count as evidence.

These studies confirm our perception that we can forget stuff.

One is a general statement of fact, the other implies introspection of one's own individual cognitive processes. I have yet to see evidence that the latter is actually possible.

The phenomenon of people forgetting stuff is not subjective. It's not one person in isolation thinking they just forgot something. Forgetting is a phenomenon on which there is a lot of social feedback and repeatable experiments.

You seem to be denying the possibility of ever connecting science back to individual perception. Denying this makes any and all science completely meaningless though.

It's not limited to observations about ourselves. You could ask the question "but aren't you hallucinating?" about absolutely everything required to verify the outcome of a physics experiment for instance.

LLMs notwithstanding, I have yet to see a replication of the human mind created by humans. Therefore, I would take any claim that neuroscience 'understands' how the human mind works (a claim that I've never heard from an actual neuroscientist) with a huge amount of salt.

Not that it matters, but for various reasons I have spent quite a lot of time around neuroscientists and I think most of them would agree that "no idea" is a pretty ridiculous way to characterize the state of the field.

> the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to “look back” at previous tokens, as opposed to traditional self-attention mechanisms. While learned time decay helps prevent the loss of information, it is mechanistically limited compared to full self- attention.

> Another limitation of this work is the increased importance of prompt engineering in comparison to standard Transformer models. The linear attention mechanism used in RWKV limits the information from the prompt that will be carried over to the model’s continuation. As a result, carefully designed prompts may be even more crucial for the model to perform well on tasks

Sadly too many folks copy and paste what works for openAI and move on when it fails

Hopefully this gets the point across without the mathematics and a precise description.

Vs

I need to memorise everything as it’s spoken out, and then answer on it

Is a good approximate on the difference.

The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training)

It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task

For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory.

And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations.

And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.

> we loaded the entire text of The Great Gatsby into Claude-Instant (72K tokens) and modified one line to say Mr. Carraway was “a software engineer that works on machine learning tooling at Anthropic.” When we asked the model to spot what was different, it responded with the correct answer in 22 seconds.

And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue

Machine learning is optimizing mathematical weights and biases, and the human mind is messy wetware. One of the first things you will read in any machine learning textbook is to throw away the notion of AI trying to simulate the human mind.

Note that what people claim here is not true: just because you go past the context window does NOT mean everything is forgotten, it means that you're back to RNN levels of performance.

In transformers, just like in RNNs, processing the nth token influences the nth token AND the (n+1)th token. So the influence of the nth token "slowly dies down". It influences the (n+1)th token a lot, the (n+2)th token a bit less and so on. The special thing about attention is that it doesn't start dying until you go past the context size.

Or with a bit of math notation. If a neural network is a function f, then:

RNN: f(x[n]) = f( f(x[n-1]), x[n] )

Transformer: f(x[n]) = f(sum(i=0...context size, f(x-i)), x[n])

The difference between LSTMs and GRUs is in the structure of f. And "sum" is, like a lot of things in here, uh, basically accurate but missing a lot of detail.

In your example, x[n-1] = f(x[n-2], x[n-1]). So really, your expression should be

RNN: x[n] = f( … f( f( f( f(k0, x[0]), x[1]), x[2]), x[3]), …, x[n-1])

where k0 is a learnable parameter.

We won't know before we've tried it. Reasoning by analogy with humans is not useful. In a year we'll have tried lots of things and can give a much better answer.

My theory is that the model will start generalising the information stored once it starts going past its limits (like real life humans)

So if we take books as an example, we probably do not remember every single word. But we might vaguely remember which section or book events occur

Combine this with the agent model, and we may have an alternative for embeddings. Where we can ask which pages the model recall is relevant to the question. Bring those pages up again. And get the answers

(Once again like how a human might answer in real life)

It’s not clear how to learn this. Transformers work so well because there are only a few layers between input tokens and output. Adding a summarization step is a whole new research project.

What would be called "attention" now, though, basically didn't exist in that system. Once you started training it on something new it lost everything.

For anyone wondering, it was written in Actionscript 3, and ridiculously, each neuron was bound to a display class that displayed as a semitransparent cube that lit up as the inputs propagated through them. A thoroughly ridiculous side project.

But other than scaling that from 1000 neurons to billions, I'm curious what has changed about the concepts of pathing or tolerance to make these models better? Maybe my concept of the principle behind modern LLMs is too archaic or rooted in a cartoon understanding of our own wetware that I tried to reproduce.

[edit: I'm describing an ancient home project... for anyone downvoting this, I'm more than receptive to hearing your reasons why it's stupid. I'm the first to admit it seems stupid!]

So keep doing weird experiments

Realastically... the problem is a data problem. The math is mostly there. Transformers et al would have been figured out in no time had we had the sort of data tools we have today. I'm talking about the ease with which you can take gigabytes of data, throw it in S3, and analyze it in minutes.

That, combined with cheap and accessible compute (cloud) and the maturation of CUDA meant it was all a matter of time before this took place.

I disagree. Even things that seem obvious in retrospect, take some time to be figured out.

Resnets, batch norm, dropout, etc are examples of this.

And I don't think transformers are obvious.

Good

- substantially cheaper and faster to run / train

- scalable to ridiculous context size

Bad

- you will need to change how you prompt this model sadly (it works differently)

https://github.com/BlinkDL/ChatRWKV

I'm a regular involved with the RWKV community.

AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time)

PS: you can find our discord here : https://discord.gg/qt9egFA7ve

One of the biggest issue for testing all of this, is it takes a crap ton of GPUs to prove all the alternatives to transformers beyond 1B param.

For example I’m waiting for someone to do a 1B-14B text based diffusion network

Finally, if this is truely the case (and all that really matter is size+dataset)

We really should use an architecture that is cheaper to train and run. And that’s what RWKV represents here

You can even run the 7B quantized model reasonably on most laptops (try the rwkv-cpp / rwkv-cpp-node project)

- it is NOT backed directly or owned by any VC funded company

- it is 100% OSS driven by the community (Apache 2 license)

- it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board

- IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not training data, so we compared similarly trained models)

And yes we do have multiple experiments and plans to make it better. It’s a list, and we will not know which is final until we try. Individual members can go to great lengths on what they are working on

For better or worse, being truly OSS means our initiatives are more disorganized then a centrally planned org

To be fair, that filters the majority of models in the scoreboard.

Where we have more OSS models to choose from without weird rule lawyering gotchas. Or needing to be from a research institute / a license to download the weights

- integrating this with AI platform X/Y/Z

- setting up evals

- improving the code quality

- making a how to guide (it’s stuck on my todo list)

- helping with dataset

- doing silly experiments on how the architecture work (and if the changes give good result)

- etc etc

One of the community goals is to make this a model for EVERYONE on earth that means we need quality dataset for all the non English languages

So even on that level there are things to do

( find something that interest you on the community )

For example, I'm currently attempting to use RWKV for named entity extraction. I ask it to analyze a piece of text and provide output in JSON format. It starts off great. However, eventually, it seems like the beginning of the JSON list 'overtakes' the question I asked, and it starts to just produce random data that would seem plausible based on the set of things in the list. I realize this is due perhaps to the precision losses of the RNN as weights decay.

However, I feel there ought to be some way we can prevent that. Any thoughts?

Ask the question / explain the task first. Then give it the data you want to extract from.

Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)

I'll say "get a list of Blah from the following document in Json format like this:

Example"

Then I feed the document and add a spot for the answer.

The model begins correctly. But usually in the middle of the Json list generation, it will veer off, and start hallucinating as if it forgot the document and the task. I'm happy to share specifics and datasets but this is a cross cutting problem.

Rwkv is able to answer my questions when I ask simple yes/no or classification. It's the listing that throws it for a loop. Transformers do not have the same problem. Both llama and gpt are able to maintain focus.

Also, do you know where I'd find information on how the current weights were trained?

(You are using raven right? That’s the instruct trained varient)

Btw ping the discord if ur looking into finetuning for your usecase

Unfortunately I really would like machine readable responses and raven is a bit too verbose.

Looking at fine-tuning right now.

Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available

Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range.

Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPUs required to make it happen

The code is already there, just need someone to run it,

Ps: I too am personally interested in how it will perform at ~60B, which I believe will be to be optimal model size for higher level of thoughts (this number is based on intuition not research)

You might find that thread interesting, they're taking submissions for potential partnership with LambdaLabs a cloud compute company that has a few hundred H100s laying around. They have an open form and their cofounder is currently doing the rounds having meetings and this may be a good candidate.

I'm not associated with them at all, just interested in the space and things going on.

8 x 8 x 8 A100, should be able to do a 100k++ tokens/s at that size

With a dataset of 1.2 trillion tokens. That’s 12 million seconds. Or 140 days

(PS: this is why everyone is training <60B, its crazy the cost, even if my math estimate is wrong by 300%, its still a crazy number)

what datasets are available to the community, how big are these, are they needed to be updated from time to time, where are these stored, what are the usual cost ranges involved, ...? :o

thank you!

If not, you are getting diminishing benefits for each param you add

I’m extreme cases your model can even perform worse with more param due to lack of training data

More complicated: the quality of the data matters as well

So there are 2 major directions. Build efficient models with good dataset and optimal param count for the task

Or go big on everything (aka openAI) which requires monster GPU time for every reply token

There are obviously in between as well. Hence why the question is so loaded

Ballpark: if your not setting aside a 100k for GPUs alone, to train a 60B model from scratch, your probably not ready to train one

Is there some kind of dedicated fund for training hardware? Donating an A100 sounds unlikely, but surely they could be crowdfunded?

If you want to help fund RWKV, the ko-fi link is - https://ko-fi.com/rwkv_lm

IMO: this needs way more funding, just to sustain blink leading this project, let alone GPUs for training.

(Also - current tests shows this model doing really badly with 4bit quantized, but alright at Q5 and Q8)

We are talking about over 10x reduction in GPU time for inferencing tokens and for training too

Aka it’s cheaper and faster

Alignment is frankly IMO purely a dataset design and training issue. And has nothing to do with the model

Just to be clear to everyone: this is “use attention to train parameters, use recurrence for inference”

It’s a very cool idea and I hope we get more interesting approaches to inference, but attention is here to stay for training.

> the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined.

And sure, this is the core concept of self attention, no?

Work on transformer alternatives, especially parallelizable ones like this, is incredibly important - it would suck if we get sucked down a local optima in architecture without actually looking at nearby viable alternatives.

Mostly lead by a single person (blink). This community consist mostly of people outside the academia / big VC tech scene

When eleutherAI offered to help us with writing the paper. Various key folks banded together for the paper, as it’s what seems to be a very strong alternative to transformers

This does not mean everyone in the discord was credited.

The requirements are for significant contributions to the paper. typically several paragraphs long worth of drafting and revisions

Just doing a line of grammar change or a single benchmark is not enough

https://arxiv.org/pdf/2305.10403.pdf

https://arxiv.org/pdf/2211.05100.pdf

That is wild, almost like film credits

https://discord.gg/qt9egFA7ve

BlinkDL has been working on this project for two-ish years, originally in the EleutherAI discord and then created his own to house the project.

I wasn’t thinking too hard about my exact wording, but yes I was thinking of EleutherAI and its various spin-off servers. EleutherAI doesn’t /run/ any of the other servers, but we all have a close collaborative relationship. I’m sure there’s a lot of duplication of membership (e.g., I’m in all of them) but quickly adding up the membership of each server comes out to around 70,000. EleutherAI and LAION are the largest at 25k each, with the others typically having around 5k each. I would expect at least 30k of those users to be unique though.

https://twitter.com/arankomatsuzaki/status/16390003799784038...

Has anyone here experimented with this recently to confirm?

We believe this can be done for 16k to way beyond 100k

Research in how RWKV handle the hidden state shows that it is barely used (imo: <5%??) meaning lots of headroom for scaling context size

(This is actively being experimented on - we dun really know the limit yet)

I'm going to take a closer look :-)

That sounds promising. Maybe scale (of model and training samples) is all you need.

And RNNs are obviously so much more efficient at inference.

I'm going to take a closer look :-)

Its main benefits being presented with the above staying closely the same, is that it has significantly lower running and training cost without performance penalty

If you want to compare any <20B model with GPT 3.5 / 4 / 100B models evals, thats another paper altogether

What's going on here?

Blink is an individual and does not represent a company (aka not google, not eleuther, not <insert VC company>, etc)

So he had to fill something up i guess haha

The idea of a foundation has been tossed around. So it may eventually happen

Also the RWKV project predates this paper by over 2 years - nothing is truely new and revealed today on this paper

Your simply just finding out about it now in a formal paper consolidating various things that was learnt in this project

I saw there is no one listed in the author list as independent researcher or no affiliation or whatever. It seems hard to believe, if it's a lot of random contributors on a discord channel. Did they all have to make up some kind of 'DBA' (doing business as) name?

There is a strong requirements for substantial contribution to the paper itself to qualify for authorship.

So unfortunately that does limit it in part to folks who are more familiar in writing such papers, have the resources for doing benchmarks, and charting, etc

And less so for folks who for eg, tinkers, contributed to dataset, porting the implementation to X lang, doing LoRa, or various other experiments that has nothing to do with the core architecture

Probably your best bet for getting a feed heads up is to set a GScholar alert on the first author?

^direct pdf link

Worst of all, they use "channel dimension" in a sequence model. What even is a channel in a sequence of tokens? This happens as soon as you have a single person with CNN background on the team and it makes zero sense. What if you actually have channels in your data? What then?

Because channel mixing, is a core component of this architecture, and that keyword "channel"is all over the place. I have no idea what is it you are critiquing specifically (i could not find the mention of "channel dimension" in the paper)

https://twitter.com/AiEleuther/status/1660811180901019648

There is RWKV in 150 lines to help understand all the nitty gritty

https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_in_150_li...

RWKV is a new language model architecture that is comparable to transformers in terms of performance. RWKV is more efficient than transformers, which makes it possible to train larger models on smaller datasets. The RWKV community is open source and welcomes contributions from anyone. There are plans to create larger versions of RWKV, but this will require more computational resources. Here are some additional details about the chinchilla law and the dataset problem:

The chinchilla law states that the amount of data required to train a language model grows exponentially with the model size. This means that it is very expensive to train large language models, even with the latest hardware. The RWKV community is working on developing new methods for training large language models more efficiently. There are a number of datasets available to the RWKV community, including:

The Pile: A massive dataset of text and code. The Chinchilla: A smaller dataset of text and code that is designed for training RWKV models. The Red Pajamas: A dataset of text and code that is being used to train a 65B RWKV model. These datasets are stored in a variety of locations, including:

The RWKV GitHub repository The Chinchilla website The Red Pajamas website The RWKV community is constantly updating the datasets and adding new ones. If you are interested in contributing, please visit the RWKV GitHub repository.

What? I though the chinchilla-optimal regime was something like "20 tokens per weight". That's not remotely exponential, even by the word's colloquial use.