First reactions before reading in detail:

Does it give more lift than just using the training data from the diffusion model? You can search through LAION 5B in Clip embedding space and pull out lots of new training data without ever using a generative model.

Are they able to use prompts to generate augmentations around whatever invariant features - different colors or positions or scenes- in order to train better?

This is the right question. It is plausible to me that this is a way of condensing a vast training set into a reduced set of high quality examples.

You could also consider some kinds of sampling of the dataset - basically not including images that are too close together in whatever space. There are also data distillation methods that are supposed to generate a smaller training set - the ones I've seen don't generate images that look like real training samples though. I think it's definitely an interesting thing to study - it would be worth exploring what specifically the generative model can add.

I've actually done something like this with GANs, but the gains came largely from generating plausible training images that were augmentations around some invariance, as I mentioned above.

is there an un-similarity search using clip. some reasonably efficient way to pull the X un-similar embeddings(to Y dataset) in Z dataset?

Un-similarity would generally just mean further away in the latent space. I don't know if there are common heuristics used for this - taking the furthest away would give you outliers, and then probably lots of members of whatever class was furthest away. So you'd probably be best first clustering in the latent space and then sampling from each cluster.

Sam Altman and Ilya Sutskever have made several mentions of the idea that models are better at teaching models than humans are, in many cases. The next GPT iteration will involve a lot of AI-teaching-AI.

It's not so simple. Garbage in garbage out. But if you can get a validation signal in there, maybe using an external tool, a game, an outcome of some sort, then maybe it can work. For example generate code, and then run it to see if it passes tests - this is a good filtering signal.

These guys did that by letting a model generate unit tests and then let the interpreter check it:

https://arxiv.org/abs/2207.14502

I don't think you necessarily need any validation signal to improve performance with synthetic data. Here's an example I posted further up:

> Suppose you want to optimize the ability for an LLM to do arithmetic. You prompt the LLM to generate a bunch of arithmetic questions, prompting it to show its working. Then you take that output, remove the intermediate steps, and train on the results.

Is there a high level explanation as to why? Best I can imagine it’s that it’s easier to improve a model than make one from scratch, and so AI training AI means you can start fresh and have the benefits of the previous version train the foundation for the next, and then yet again refine that new version and repeat.

… but this is 100% speculation, I have no clue. Thoughts?

The task of generating images is harder than that of classifying, the training signal is also much denser. So it means the diffusion model implicitly learned a better visual model, and now they found a way to invert that and improve the discriminative task as well.

We can do the same thing in text. A recent paper showed GPT-4 is better than most human labellers at tagging NLP datasets. I personally used it to generate samples for smaller models, basically the same thing this paper showed. You can even top that with a second LLM round to reflect on its generated data and filter the noisy examples.

In the end it still makes errors. And the errors GPT makes are hard to detect at scale.

Can you give an outline of how you used gpt4 to generate samples?

Probably because you can use the error gradients from the generative model somehow

Alan Turing actually talked about this way back in Computing Machinery and Intelligence and it’s already used for distillation of information from large models to more compact ones in the so called “student/teacher” training regime.

isn't reinforcement learning from human feedback (RLHF) the process of training a model for a reward signal from humans?

This is one of the reasons why we are probably not going to "run out of tokens" to push transformers to the next level. And we haven't even started really using visual data as inputs to train auto-regressive multi-modal LLMs.

There's no free lunch when it comes to learning signal. The right comparison here is between training on synthetic data and training on the data used to create the synthetic data generator.

Synthetic data doesn't create more learning signal, it just creates more data repeating the same learning signal with some bias assumptions that might as well be in your final model.

The difference is moreso in training times for retraining new models. If you've taken highly disparate data and compressed it into a particularly "juicy" dataset which is smaller, that could bring down training times.

That is to say, it's related to curriculum learning moreso than breaking information theory.

If a model truly generalizes, why wouldn't it be possible for it to generate more real signal than was used for training it?

Isn't that the whole point of using models instead of, I don't know, huge lookup tables?

The theory behind models is that they learn the distribution used to “generate” the data. A good model has a good approximation of the distribution. But then anything generated from it follows the approximation of the distribution not the distribution itself I.e. the generated data is not going to be good training data for the generating model.

You can for example train a very large and good model and use that to generate more data to train a smaller model e.g. for a faster inference use case. That data follows a closer approximation of the underlying distribution than the small model has so it can still be used for convergence to the underlying distribution.

AlphaGo Zero was trained on synthetic data which it created itself, it learned from self-play. So why wouldn't synthetic data work for GPTs? I guess a difference is that AlphaGo Zero uses reinforcement learning with objective reward signals (such as winning the game), while GPTs perform unsupervised learning (imitating the training text) without such a signal. There is no signal which tells the model whether the text it generated was or wasn't close to the real distribution.

For some types of language model this signal could be generated though. If we have a language model which is able to generate pairs of the form (mathematical conjecture, proof attempt) in a formal language, then an automatic proof checker could generate the reward signal, i.e. proof correct / incorrect. The difficulty is probably to get the process bootstrapped since you need a certain amount of base proving capability to get the ball rolling.

You’re confusing two different concepts. AlphaGo learns a distribution where given a game state it generates a move that maximises its internal probability of victory. There is a second “distribution” namely that any terminated sequence of go moves has an objective result.

AlphaGo samples from the latter distribution in a guided way (as the space of all go games is computationally intractable). It uses its learned distribution to do that guided sampling and uses the objective outcomes of the known distribution to inform its own learned distribution.

One way to think about this in the context of language modelling. Suppose I want to build a language model that says the word “goal” at least once every 2000 tokens generated. I could then repeatedly generate from the model and objectively score whether it has generated that word or not in each occurrence (the analogy of the finished go game). I then can use this objective scoring function to compete models against each other and do the alpha go style training. You can see here how the new training data is sampled from a different distribution than just regular language.

Your "goal" example sounds like a more useless version of my theorem prover AI?

>> If a model truly generalizes, why wouldn't it be possible for it to generate more real signal than was used for training it?

If by "a model [that] truly generalizes" you mean one that generalises outside its training distribution, then neural nets can't train such models. I don't reckon any machine learning approach can do that.

In any case this is ImageNet we're talking about that's been done to death many times over already. Far from learning more general models, after a certain point in time improvements in accuracy have meant that models are getting better at overfitting. Same with MNIST, where that happened a long time ago.

Obligatory reference to defend against this comment being knee-jerked to oblivion:

>> This stands in sharp contrast with what deep nets do, which I would call "local generalization": the mapping from inputs to outputs performed by deep nets quickly stops making sense if new inputs differ even slightly from what they saw at training time.

Francois Chollet, The limitations of deep learning

https://blog.keras.io/the-limitations-of-deep-learning.html

From the linked blog post (2017)

> Even with this data, you could not train a deep learning model to simply read a product description and generate the appropriate codebase.

Would be make the same claim now I wonder?

Sorry, I can't edit my earlier comment but I re-read our conversation and I realise I misread your quote. In particular:

>> read a product description and generate the appropriate codebase

Chollet here is talking about creating an entire application, not short code snippets or larger functions. Producing short programs from specifications was already possible when Chollet wrote the above-linked article. It is still not possible to genera the entire codebase for an application by reading a product description. Not anywhere near that.

Agreed, but the question is would he still claim it's not possible to train a deep model to achieve this goal at all. Maybe not, but it feels a lot less certain than it did in 2017.

I wonder too. I've communicated with Chollet in the past (about his ARC dataset) and he's responsive. You could email him and ask, or you could ask him on his twitter (I don't use twitter).

I genuinely don't see how this viewpoint is compatible with the grokking results.

I'm sorry, I'm not familiar with the grokking results. What are you referring to?

This comment reminds me of a time, many years ago, when someone wanted an the internal design team to make a big poster out of a low res image.

The design team explained that they couldn't blow the image up to the size of a poster because it would look too grainy as it was extremely low resolution.

The individual demanding the poster replied with a 'brilliant' solution: "Why don't you just take a picture of the low-res image, and then use that hi-resolution picture to blow up the image!"

The same applies here. You need more information to learn the signal better, by definition there is no new information in the model.

> The same applies here.

Does it, really? Analogies are great for explaining ideas, but aren't that great as logical arguments.

> You need more information to learn the signal better

You need more information than there is in the training set - which a well-generalizing model is supposed to be able to generate. Yes, there may be some bias if model doesn't generalize well, but that's why I put it in the assumption...

> Does it, really?

Yes. It's information theory.

You can add information, by making rules like the class doesn't change if the thing is a different color or size or position, or on a different background. But you can't automatically create new images from a distribution that has been learned and expect them to add information.

Then how do humans do it? Our knowledge relies heavily on sending the output of humans to the input of humans (verbal communication, books, self-thought, etc). At one point total human knowledge was little more than "some berries are bad" but through this recursive process arrived at the rich garden of information we have today.

Because humans (and your language is confusing here between individual humans as models and human society as the model) are constantly receiving new information. Even before humans learned "some berries are bad" they initially thought either "all known eaten berries are bad" or "all known eaten berries are good", then somebody at a good/bad berry and updated their believes (i.e. their model of the world).

Humans learn from other humans because humans don't individually share the same information and model of the world. A science teach can speed up how you learn science by taking the compressed information and explaining it quickly (essentially what is happening in the post), but out scientific model is expanded when we have experiences that call into question the strength of our current model.

However both individually and as a society we are constantly taking in new information (sometimes more sometimes less) and using that to update our model.

Humans don't just try to imitate text from some group of humans, like language models. But if we wanted to imitate the text of, say, 18th century novelists, then only those 18th century novels seem to matter. If we also started to imitate our own imitations of 18th century novels (synthetic data) the result would probably get worse, since our imitations are imperfect.

To consider your analogy a bit further: Imagine you had access to only... five 18th century novels, that's it, no other knowledge about the period, the people or anything. Then you try to write your own. You can make up your own stories that are a merging of various parts of those five novels, but they'll always only contain what's in those, or some combination.

If all the books deal with France then Germans or English will never make an appearance because it's literally impossible to guess that they exist since they're not in the original books (the books here are training data if that's not obvious).

Humans have nothing in common with neural networks (other than if you choose to misunderstand the literal definition of those two words), there's no direct analogy to be made.

I didn't say anything about neural networks, I asked how humans do it.

To start with, humans have access to far more data than just what is generated by other humans. We can look at nature, other animals, mathematical laws and many other things.

Humans also have far more ability than current model architectures to use memory (particularly in training), flexible context (Model context always directly follows the input distribution in training whereas we can flexibly combine different parts of our input data as we like ) and logic. We can also do live learning in ways current models can’t.

Here's a scenario where training on the output of a model could improve its own performance:

Suppose you want to optimize the ability for an LLM to do arithmetic. You prompt the LLM to generate a bunch of arithmetic questions, prompting it to show its working. Then you take that output, remove the intermediate steps, and train on the results.

I think you would improve some benchmarks by doing this, possibly at the expense of others.

I'm sure you could do similar things with other types of model, but I think thissimple example shows the point adaquately.

In any case, information theory is not especially relevant here. It puts an upper bound on model performance but we are so incredibly far away from that bound it isn't worth worrying about.

> You prompt the LLM to generate a bunch of arithmetic questions, prompting it to show its working. Then you take that output, remove the intermediate steps, and train on the results.

Removing or editing the output of the model is providing new information to the model, what's improving the performance of the model in this scenario is that you are explicitly adding new information and fine tuning it on these new cases.

> information theory is not especially relevant here

It's extremely relevant because people seem to be arguing with about mathematical facts as though they were somehow opinions.

You cannot improve the performance of a model without adding new information to that model.

What information is being provided? I'm not suggesting that you would do any kind of check to see if the new training data is correct. Just take the output and remove every line between the first and last.

Deterministically removing data is not adding information, unless you are defining information in an extremely unusual way.

(Not the OP) What I wonder about is why would performance increase with what you suggest. In my mind, you're proposing getting a language model to generate its own training data. But data on its own is not enough to train a good model. The data has to have some kind of information content that guides the learning algorithm to select a good model. If you feed the training algorithm data that doesn't increase its information about what a good model is, you're not helping it train a better model.

For the record, I tried something like what you suggest when I was doing my Master's. That was back in 2014, and I had to train a classifier for a machine learning class. I was given a training set while a separate validation set and test set were kept private (it was all set up in Kaggle as a private competition). To clarify, the idea was that you trained your classifier of choice on the training set, then labelled the validation set with your trained model and submitted the labelling to get a score that you could use to improve your model. The last day of the competition you had to make a choice and submit a final model, that would be evaluated on the test set, for which you had no information.

The problem was that the training data was not very much. There was more data in the validation set, but the data in the validation set wasn't labelled. So I tried to label the validation set with a model I trained on the training set. And, what would you know. My classifier scored 100% accuracy on the validation set. But when I submitted my trained model on the test set it did much worse, I think close to 60% or so.

Empirically demonstrated then: you can't dogfood a classifier to a better version of itself. When you train a classifier on some data, the classifier learns the underlying distribution of the data, with some amount of error. If you then label new data with the trained classifier and retrain the classifier on its own labelling, you end up multiplying the error.

Btw, that doesn't change with language models, large or small, and it doesn't make a difference whether the model has an unsupervised training step or not. As long as your model is, well, modelling, some unknown true distribution and incurring some error, reusing the trained model to generate new data will generate data with error.

So I don't think what you say can work and I'm curious to understand why you think it will. What are you saying will happen, exactly, if you dogfood an LLM's generations, like you suggest, that will make it improve?

> You can add information, by making rules like the class doesn't change if the thing is a different color or size or position, or on a different background. But you can't automatically create new images from a distribution that has been learned and expect them to add information.

I am not quite sure I follow. Why would any of those things be true for all models?

How does this train of thought apply to something like AlphaZero? There, computation/time seems to be spent creating data. Why can't the same thing in concept apply here?

Good question. I'm not an RL expert, but those kind of models are sampling the possible space of game states, based on the rules of the game, to learn a winning policy. Complex games can have effectively infinite different states.

What we're talking about here is approximating a distribution from a finite and comparatively small set of images. There's no "game" you can play to get new images, the training data is fixed.

The analogy might be say sampling images from a computer animation, which shows the object your interested in in different plausible poses and backgrounds, lighting, angles. There's a question of making sure this generalizes to "real" images, but that can help. But it relies on the animators knowledge of what the object can look like in different poses. That's how information gets added.

> Analogies are great for explaining ideas, but aren't that great as logical arguments.

In this case it is hardly an analogy. With respect to Information Theory, compression and machine learning are essentially the same thing [0]. If you can understand why taking a hi-res photograph of a lo-res image does not create more information (and therefore cannot be used to make a hi-resolution poster) then it's essentially the exact same logic explaining why outputs of a model cannot be used to improve that same model.

More to this point, which other comments have already pointed out, what's interesting about this post is it asks whether or not very, very large data sets can be reasonably compressed by models such that they can simulate training another model while requiring less data.

From an information theoretic perspective this latter case is both completely sound and of potentially big practical importance.

I do have to say, I'm a little surprised that basic information theory is no longer common knowledge among the HN community (more-so surprised at a resistance to it), especially among people interested in Machine Learning. Out of curiosity, do you have a background in ML and/or Comp Sci?

0. https://en.wikipedia.org/wiki/Data_compression#Machine_learn...

True, with the connotation that architectur and preprocessing pipeline are kept equal. E.g. if you're training convnets, data augmentation like rotating and flipping increases generalization.

checking correctness of things like factorization is easier than factoring in the first place.

I had an idea yesterday, why don't we train AI on all the chats, like have a rolling release, where we feed it's chats or maybe even a self-retrospection of the chats back into the thing with semantics around quality, so it can improve that way?

Example: You ask ai something, it takes 3 different tries to get what you want. AI Critiques itself, decides what it could do better and rolls that into training, perhaps it goes line by line and scores how 'helpful' that was to give certain negative/positive bias to new training datasets.

I believe Tesla uses synthetic data for FSD/Autopilot training as well. Not sure what's the ratio between synthetic data and real videos, but it seems that they plan to rely more onto the first group in the future.

https://www.youtube.com/live/j0z4FweCy4M?feature=share&t=571...

My immediate thoughts:

1. The human brain must be doing something analogous to this, continuously, to learn from a relatively small numbers of samples.

2. Going forward, generating high-quality synthetic data looks likely to become a standard practice for training models.

3. Whoever has the largest, highest-quality synthetically generated datasets will have the best-performing AI models.

Re (3) I almost think it implies the opposite: if everyone has access to synthetic data from large foundation models, it will be those that have their own proprietary data that can stand out. This is already true imo. There are lots of competing models with similar performance on imagenet, but when you get into niche computer vision, data is the main differentiator.

If, as you write, "everyone has access to synthetic data from large foundation models," then I'd agree. But I expect we're going to see a transition in the opposite direction, with more corporations keeping newer models and data proprietary to maximize competitive advantage. I suspect this kind of thinking factored into OpenAI's decision not to release any details about GPT-4.

Fair enough. I think that many would like to have their own proprietary models, but for now, these "web scale" datasets really only come from scraping the internet. Companies can do their own cleaning and whatnot, but the data distribution still largely reflects what's on the internet because that's all there is.

To your point though, three are definitely some proprietary pockets, like the Shutterstock/openai thing. Or the various RL layers ala chatgpt. But a lot of the value there is in the labeling, and I expect we'll see more competing versions that are open (we already are)

I hope you're right :-)

You mean like dreaming?

Training on synthetic data has the problem that the models may drift away from the original distribution because errors could get amplified.

Exciting news, thanks for sharing! We've been applying this technique to create custom models on the fly with our no code platform at https://remyx.ai We're trying to build up our user base and get more feedback, try it out or check out our walkthrough: https://youtu.be/7SMySnRRTew?t=39

seems logical in that you can train on more poses/colors/shapes/angles/lighting conditions. but also we could assume that there's a more direct/elegant improvement than using a generative model?