 |
ML Newsnew | past | best | rss | submit | about |
| Single Headed Attention RNN: Stop Thinking with Your Head (arxiv.org) | |
| 3 points by nl on Nov 27, 2019 | hide | past | pdf | 1 comment on HN | |
|
In plain words: An older text model with built-in memory gets one simple attention step, letting it cheaply look back over earlier text. Trained on a single desktop GPU in a day, it nearly matches the best systems that predict text one character at a time. Abstract · Single Headed Attention RNN: Stop Thinking With Your HeadThe leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street. Transformers this, Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer scale silicon. We opt for the lazy path of old and proven techniques with a fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result. We take a previously strong language model based only on boring LSTMs and get it to within a stone's throw of a stone's throw of state-of-the-art byte level language model results on enwik8. This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer. The final results are achievable in plus or minus 24 hours on a single GPU as the author is impatient. The attention mechanism is also readily extended to large contexts with minimal computation. Take that Sesame Street. | |
| add comment on HN |
We also achieve state-of-the-art on WikiText-103 - or do we? This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer. The final results are achievable in plus or minus 24 hours on a single GPU as the author is impatient. The attention mechanism is also readily extended to large contexts and requires minimal computation. Take that Sesame Street.
Many fight against the homogenization of language by dividing and conquering as they did in the Tower of Babel era (see:Javascript frameworks).
Perhaps the reserchers give up on machine learning and in-stead decide to pursue a musical career themselves? Who am I todictate what they do in this alternate timeline?