ML Newsnew | past | best | rss | submit about
Transformer models: an introduction and catalog (arxiv.org)
188 points by mariuz on Feb 16, 2023 | hide | past | pdf | 25 comments on HN

In plain words: A plain-language guide and catalog that sorts popular Transformer-based AI models (the design behind ChatGPT) and explains their key ideas. It covers models trained on raw text alone as well as those further trained with human feedback, and explains what each name means.

Abstract

In the past few years we have seen the meteoric appearance of dozens of foundation models of the Transformer family, all of which have memorable and sometimes funny, but not self-explanatory, names. The goal of this paper is to offer a somewhat comprehensive but simple catalog and classification of the most popular Transformer models. The paper also includes an introduction to the most important aspects and innovations in Transformer models. Our catalog will include models that are trained using self-supervised learning (e.g., BERT or GPT3) as well as those that are further trained using a human-in-the-loop (e.g. the InstructGPT model used by ChatGPT).

Xavier Amatriain, Ananth Sankar, Jie Bing, Praveen Kumar Bodigutla, Timothy J. Hazen, Michaeel Kazi
arXiv:2302.07730 · cs.CL · submitted Feb 12, 2023 · updated Mar 31, 2024
abstract · pdf · html

add comment on HN

xamat on Feb 16, 2023 [–]
Thanks for the feedback everyone. Here are the sources in case anyone wants to contribute (or fork): https://github.com/xamat/TransformerCatalog
reply
adamnemecek on Feb 16, 2023 [–]
I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra.

https://arxiv.org/abs/2302.01834v1

The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback.

I argue that autodiff can be replaced with what I call in the paper Hopf coherence.

Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.

I'm working on a next gen Hopf algebra based machine learning framework.

Join my discord if you want to discuss this further https://discord.gg/mr9TAhpyBW

reply
amkkma on Feb 17, 2023 | parent [–]
Is there any hope of understanding this with just calc and linalg knowledge?
reply
adamnemecek on Feb 17, 2023 | parent [–]
I think so. The main idea is the idea of Hopf coherence. The transformer/Hopf algebra update their internal state in order to enforce the Hopf coherence formula (you can find that in the paper).

The idea of streams (as in infinite lists) is related to this via coalgebras.

reply
erichocean on Feb 16, 2023 | parent [–]
> Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.

Have you written any more about this?

reply
adamnemecek on Feb 16, 2023 | parent [–]
Look into the connection between diffusion and Hopf algebras.
reply
mi_ainsel on Feb 17, 2023 | parent [–]
Apologies if I'm missing something completely obvious but I'm having some difficulties dredging up material that connects Hopf algebras and diffusion. Are there any resources that you would recommend in addition to your paper?
reply
erichocean on Feb 20, 2023 | parent [–]
Hi Adam, can you update your Discord invite? It's now invalid.
reply
amatecha on Feb 17, 2023 [–]
I was so certain this was discussing Transformers like, the action figures, and have never been so confused looking at both a link and the comments section on HN before. Especially considering: https://github.com/xamat/TransformerCatalog/blob/main/02-01.... I'm just going to keep scrolling now :'D
reply
visarga on Feb 17, 2023 [–]
> 2.5.9 ChatGPT is also more than a model since it includes extensions for Memory Store and retrieval similar to BlenderBot3

I don't think this affirmation is factual. There are people who played with this idea, but it is not part of chatGPT.

reply
sva_ on Feb 17, 2023 [–]
> 2.5.5 BERT

> Extension:It can be seen as a generalization of BERT and GPT in that it combines ideas from both in the encoder and decoder

I believe this is an error? Text from BART. And a space missing.

reply
visarga on Feb 17, 2023 | parent [–]
I have a hunch they used LLM to compile the list.
reply
DerSaidin on Feb 17, 2023 [–]
It is a shame the figures 5,6,7,8 break up the content of 2.5 Catalog, just to fit the figures onto pages.

Are pages even needed anymore?

reply
h_lezzaik on Feb 17, 2023 [–]
Good timing, I've been trying to compile a list like this myself to keep track of everything released.
reply
theredlancer on Feb 16, 2023 [–]
Where's Cliffjumper and Ironside?
reply
zndr on Feb 16, 2023 | parent [–]
I'm glad I'm not the only one looking for a taxonomy of refugees from the great Cybertron wars
reply
sircastor on Feb 17, 2023 | parent [–]
When I was younger I would often encounter mentions of electrical transformers, and be quite disappointed when it wasn't related to the toys or the series. Even in my 40s I still have a bit of disappointment about it...
reply
robertlagrant on Feb 17, 2023 | parent [–]
Don't get me started on when we learned about the Terminator on the moon.
reply
peresthe on Feb 16, 2023 [–]
"The goal of this paper is to offer a somewhat comprehensive but simple catalog and classification of the most popular Transformer models."

Yet of the 6 comments here, 2 of them are complaining about missing models and three more are arguing about the typesetting on figures.

reply
pama on Feb 16, 2023 | parent [–]
Per the paper, here is the link to the data in fig 5: https://docs.google.com/spreadsheets/d/1ltyrAB6BL29cOv2fSpNQ...
reply
swyx on Feb 16, 2023 [–]
figure 5 on page 10 is a ridiculously small font and unreadable. i wish there was a better way to display this kind of info on PDFs
reply
mdp2021 on Feb 16, 2023 | parent [–]
> i wish there was a better way to display this kind of info on PDFs

...Portable Document Format was /born/ to display vector (i.e. you just zoom in)... The error in the page was to embed a raster image of text!

reply
nighthawk454 on Feb 17, 2023 | parent [–]
Another tip is arXiv lets you download the original tar file from the "Other Sources" section with all the original images etc in it.

https://arxiv.org/format/2302.07730

reply
dylan604 on Feb 16, 2023 | parent [–]
From the bottom of the page in question: "Figure 5: You can access the original table at https://docs.google.com/spreadsheets/d/ 1ltyrAB6BL29cOv2fSpNQnnq2vbX8UrHl47d7FkIf6t4 for easier browsing across the different model features."
reply
abc20230215 on Feb 16, 2023 [–]
Does not even list Lundahl transformers...
reply
About | RSS | RSS (all) | HN arXiv