https://arxiv.org/abs/2308.14711

An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training.

A branch is computed as branch(input, N), with a neural network N computing a scalar c=N(input), then using a sigmoid to do a soft branch by returning the weighted sum of the recursive call s(c)*branch(input, N_left) + (1-s(c)) * branch(input, N_right) (the two weights s(c) and 1-s(c) sum to 1). They only do "proper processing" using the leaf nodes.

Then they add a new loss term that encourages hard decisions by minimising the entropy of the Bernoulli distribution, making the 2 weights converge to 0 and 1, at which point only one branch needs to be taken at inference. They also state that this hardening often happens automatically though.

It's a simple idea but the loss formulation is nice, you usually want your loss terms to be a measure of information.

This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power.

Is this only for inference though? could it boost training?

Hold on, you don't even need to freeze the branches completely: each node could train 1 branch on the path to its leaf node and communicate a change in the branch node to a central server, so you can distribute training without having to pre-freeze the branches. Still would need some pre-training though, and the splits would change slowly, and the attention mechanism could complicate things.

Currently distributed neural network training SETI@HOME style looks like a complete pipe dream that nobody is taking seriously. But a smart branching mechanism like this could suddenly make it possible. Folding@home reached 1.5 exaflops, which made it the world's largest supercomputer. Imagine the models we could train that way, they would far surpass whatever OpenAI or Google could train and would be public.

[1] https://github.com/learning-at-home/hivemind [2] https://github.com/bigscience-workshop/petals [3] https://chat.petals.dev/

Well, I'm assuming they'd use whatever they're given, so maybe the question should be "how much less time would training take on a 1.5 exaflops computer?"

A lot of clusters are totally homogeneous, at least within some very large domains, so for a given interconnect and a generation of GPU you know the maximum message latency, the peak sustained pflop rate, and so on but what often matters is some combination of the depreciation-cost-per-time and the watt hours per unit time, where you can sort of approximate both if you ignore the unfortunate realities, which then act as a multiplier.

For example, a problem is network issues - and not just scale - as the training sequence often involve billions of cycles of short compute-sync sequences which are bursty (e.g., all-to-all, barrier, compute, barrier, all to all, ...) but between which there isn't enough time to engage low power modes so you're burning $ due to slack and waste. This is true in different ways for a lot of training approaches.

You can approximate this, but it's so sensitive to data set size, specific training schedule, etc. that you won't be able to get the most important answer.

If this becomes true then it’s a game changer. I hope you are correct.

Does this also mean that it would be possible to train on parallel GPU-poor setup instead of needing lots of GPU memory / bandwidth on one computer?

(1): As a byproduct, the learned regions can also be used as a partition of the input space for interpretability, surgical model editing, catastrophic forgetting mitigation, reduction of replay data budget, etc..

It's easy to get hyped up on the big tech labs because they have the most compute, but the best papers come from smaller labs and unfortunately more lately face larger challenges in getting published. It's the smaller works that create the foundations that end up in these giant models. ML is in a really weird space right now.

DiLoCo: Distributed Low-Communication Training of Language Models - https://arxiv.org/pdf/2311.08105.pdf

From the first author on Twitter: "It could quite a big deal for people who don't have access to a colocated cluster of GPUs:

e.g. with DiLoCo you could train your model, with data-parallelism, across all GPU providers, looking in real-time for the cheapest price, even if pre-emptable, even across continents"

https://twitter.com/Ar_Douillard/status/1724839420820361687

(Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are)

To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.

I don't think this was some conspiracy by big labs to push back against us (we're nobodies) but rather that people get caught up in hype and reviewers are lazy and incentivized to reject. You're trained to be critical of works and especially consider that post hoc most solutions appear far simpler than they actually are. But context matters because if you don't approach every paper with nuance it's easy to say "oh, it's just x." But if those ideas were so simple and obvious they would also be prolific. I see a lot of small labs suffer the same fate simply due to lack of compute. If you don't make your new technique work on many datasets it becomes the easiest thing to reject a paper by. ACs aren't checking that reviews are reasonable. I've even argued with fellow reviewers about papers in workshops -- papers I would have accepted in the main conference -- that are brushed off and the reviewers admit in their reviews that they do not work on these topics. I don't understand what's going on but at times it feels like a collective madness. A 10 page paper with 4 very different datasets that solves a problem, is clearly written, has no major flaws, and is useful to the community should not need defending when submitted to a workshop just because reviewers aren't qualified to review the work (this paper got in btw). We are moving into a "pay to play" ecosystem and that will only create bad science due to group think. (another aspect of "pay to play" is in the tuning. Spending $1M to tune your model to be the best doesn't mean it is better than a model that could not afford the search. Often more than half of resources are spent on tuning now)

As to paper reading, my suggestion is to just start. This is a fear I faced when I began grad school and it feels overwhelming and like everyone is leagues ahead of you and you have no idea where to begin. I promise that is not the case. Start anywhere, it is okay, as where you end up will not matter too much on where you begin. Mentors help, but they aren't necessary if you have dedication. As you read you will become accustomed to the language and start to understand the "lore." I highly suggest following topics you find interesting backwards through time, as this has been one of the most beneficial practices in my learning. I still find revisiting some old works reveals many hidden gems that were forgotten. Plus, they'll be easier to read! Yes, you will have to reread many of those works later, as you mature your knowledge, but that is not a bad thing. You will come with newer eyes. Your goal should be to first understand the motivation/lore, so do not worry if you do not understand all the details. You will learn a lot through immersion. It is perfectly okay if you barely understand a work when first starting because a mistake many people make (including a lot of researchers!) is that a paper is not and cannot be self contained. You cannot truthfully read a work without understanding its history and that only comes with time and experience. Never forget this aspect; it is all too easy to deceive yourself that things are simpler than they are (the curse of hindsight).

I'd also suggest to just get building. To learn physics you must do physics problems. To learn ML you must build ML systems. There are no shortcuts but progress is faster than it looks. There's hundreds of tutorials out there and most are absolute garbage but I also don't have something I can point to that's comprehensive. Just keep in mind that you're always learning and so are the people writing tutorials. I'm going to kinda just dump some links, they aren't in any particular order sorry haha. Its far from comprehensive, but this should help you getting started, nothing in here is too advanced. If it looks complicated, spend more time, you'll get it. It's normal if it doesn't click right away and there's nothing wrong with that.

https://www.youtube.com/@Mutual_Information

https://www.youtube.com/@EmergentGarden

https://www.youtube.com/@pascalpoupart3507

https://www.youtube.com/@AndrejKarpathy

https://www.youtube.com/@alfcnz

https://www.youtube.com/@rmcelreath

http://neuralnetworksanddeeplearning.com/

https://adversarial-ml-tutorial.org/introduction/

https://www.deeplearningbook.org/

https://nlp.seas.harvard.edu/2018/04/03/attention.html

https://huggingface.co/blog/annotated-diffusion

https://lilianweng.github.io

https://pytorch.org/ecosystem/

https://medium.com/pytorch/archive

https://www.inference.vc/

edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't

""" One may ask whether the conditionality introduced by the use of CMM does not make FFFs incompatible with the processes and hardware already in place for dense matrix multiplication and deep learning more broadly. In short, the answer is “No, it does not, save for some increased caching complexity." """

It's hard to beat the hardware lottery!

> We therefore leave the attention layers untouched

Meaning, presumably, that the GPU memory remains the bottleneck

Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

I'm really interested, can you share where you got these numbers?

8-bit ops, inference only, low memory embedded, excluding the host, implied utilization from FPS specs is ~20%

But the trend is there.

There are also newer ADAS/AV units from China which claim 1000tflops and cant really cost more than $1000/$2000 per car.

These are all tiled designed (see also dojo/tesla) heavily over-weighed on flops vs memory

[1] https://www.axelera.ai/

[2] https://hailo.ai/

H100 is basically ~$2/(2000 tflops/s)/hour or $1 for 4*10^18 floating point operations.

mind blown

I think it likely depends on the use case, but many llama models can be returned and there are quite literally thousands of free versions available.

HuggingFace: https://huggingface.co/openchat/openchat_3.5

On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.

    What is the sum of odd numbers in this set: 12345654321, 123456543212, 123456543213, 12345654324? Output only the sum, no code.

    The sum of the odd numbers in the set {12345654321, 123456543212, 123456543213, 12345654324} is 246913086434.

>>> (-1 + 2 + 9 * 5) =

... (-1 + 2 + 9 * 5) = 43

[1] https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main

https://huggingface.co/pbelcak/UltraFastBERT-1x11-long

I guess this means it isn't pretrained yet? Is it still just random weights?

It doesn't mean that there's anything technically wrong with the language model per se as a model of language, just that there has been no effort made to ensure it's fit to be deployed as-is for any given generative-AI use case, and the model authors would prefer you didn't do that.

As far as I can tell you can take Llama 2 modelling code, training infrastructure, training data and apply proposed modification (they provide PyTorch nn.Module which should be drop in replacement of nn.Linear) and run the training if you have enough compute and it should work. Doesn't mean it would work, there are always lots of practical problems, but it should work in principle.

----

    L (Number of Layers): 12 transformer blocks.

    H (Hidden Size): 768 units in the hidden layers.
  
    A (Number of Attention Heads): 12 attention heads.

     WordPiece Embeddings: 768 (hidden size) * 30,522 (vocab size) = 23,440,896 parameters.
        
     Positional Embeddings: 768 * 512 (max sequence length) = 393,216 parameters.
     
     Segment Embeddings: 768 * 2 (number of segments) = 1,536 parameters.
  
     Total Embedding Parameters: 23,440,896 + 393,216 + 1,536 = 23,835,648 parameters.

   Each transformer block has the following components:
   
       Self-Attention Layer: Each attention head has 768 / 12 = 64 units.
  
            Query (Q), Key (K), Value (V) matrices: 3 * (64 * 768) = 147,456 parameters per head.
 
            Across 12 heads: 147,456 * 12 = 1,769,472 parameters.

            Output layer of the attention mechanism: 768 * 768 = 589,824 parameters.
       
      Feed-Forward Network (FFN):
      
         First layer: 768 (input) * 3,072 (intermediate size) = 2,359,296 parameters.
   
         Second layer: 3,072 * 768 = 2,359,296 parameters.

            Total FFN parameters per block: 2,359,296 + 2,359,296 = 4,718,592 parameters. -----------------> *This is the number to keep in mind.*
        
     Total Parameters per Block: 1,769,472 (self-attention) + 589,824 (output) + 4,718,592 (FFN) = 7,077,888 parameters.
        
     Total for 12 Blocks: 7,077,888 * 12 = 84,934,656 parameters.

    Layer Norm and Other Parameters:
        
        Each transformer block also includes layer normalization and other small components, which add a relatively small number of parameters.

        Embeddings: 23,835,648

        Transformer Blocks: 84,934,656

        Layer Norm and Others: A small number, completing the total to around 110 million.

4.718M FF Params per block * 12 ~ 56.6 Million/110M Params which is a staggering ~50% reduction in size at inference time if you use 0.3% of the FF neurons for FFF??

some of which is from the pytorch docs here: https://pytorch.org/tutorials/intermediate/torch_compile_tut..., e.g. the `timed` function and how they generate data.

Also its not just the same 12 neurons, its the 12 neurons based on the previous dot product. So some kind of JIT is needed to load the right ones?

https://github.com/princeton-nlp/tree-of-thought-llm

My experience has been that most CPU inference is actually not compute limited, but memory bandwidth limited, since most weights are used for a few operations per token (how quickly can you load and unload the entire 70 GB of weights into your registers?). It's not quite that bad but I found most vectorization changes didn't meaningfully change performance.

I'm also curious if this model architecture would achieve the grokking of more complex concepts at scale.

Why is the context size and batch size represented as a single parameter?

I wonder if the conditional in this would hurt performance at scale

> Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by replacing feedforward networks with fast feedforward networks (FFFs). While no truly efficient implementation currently exists to unlock the full acceleration potential of conditional neural execution, we provide high-level CPU code achieving 78x speedup over the optimized baseline feedforward implementation, and a PyTorch implementation delivering 40x speedup over the equivalent batched feedforward inference. We publish our training code, benchmarking setup, and model weights.

Conclusions

> We present UltraFastBERT, a modified version of the (crammed)BERT architecture that uses fast feedforward instead of feedforward networks in its intermediate layers. UltraFastBERT serves as proof that large language models only really need to engage an exponential fraction of their parameters to perform individual inferences. UltraFastBERT-1x11, our deepest model with the highest promise of acceleration, uses only 0.3% of its neurons during inference and already achieves a 78x CPU speedup over the inference time of the corresponding feedforward layer. With a theoretical speedup promise of 341x at the scale of BERT-base models, we hope that our work will inspire an effort to implement primitives for conditional neural execution as a part of device programming interfaces.

So long as I've been involved with neural networks for text analysis it's seemed to me that we really should be using sparse activations because any particular document only involves a limited set of concepts.

For instance a search engine for patents might be looking at a patent for adhesive tape which activates a certain set of concepts but is not going to activate concepts involved with bicycle derailleurs or public key cryptography: a sparse representation reflects this and dense representations don't.

In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.

Does this mean everyone could be running the 100+b models from ram?

This opens up a lot , some models could be run very fast on small machines with this.

Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.

That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.

Does this not indicate a 40x speedup on the GPU?

Edit: looking at the paper, their "Naive CUDA" implementation also shows a 117x speedup in Table 2.

I know HN can sometimes be the place where humor goes to die, but I found this comment hilarious.

But in reality, sparse NN is just loose it's performance, mean loose precision and recall. Precision, means, larger probability of errors; recall - if you work with piece of information, which could consist of few, ie predicates, it will see not all predicates.

To be concrete, for good trained full-scale NN, usually considered 70-90% for precision and for recall; but if use small fraction of weights, usually will got drop of performance to about 40-70%, which is good enough for many cases, considering saves on size and computations.

All valuable AI research is secret now, they just churn out papers to waste time

We are creating a monster.

This TED talk articulates it well: https://youtu.be/WX_vN1QYgmE

What is after capitalism?