Anyone know any research combining RNN like architecture to transformers? It’d be neat if a layer of a transformer could decide to loop its output vector to a previous layer n number of times to allow it to “think harder”. The implementation would be tricky to get right though.

I've been working on using spiking neural networks (highly recurrent) to directly transform text via buffer mechanisms actuated by spiking activity. Ive had some marginal degree of success but there seems to be a bit of a wall. In these experiments, "thinking harder" takes the form of a growing pending spike queue. For practical reasons, this activity needs to be clipped or inhibited at some point. In real world biology, I suspect the power law growth in spiking activity is what contributes to much of cognition/intelligence. There's really no practical way to simulate it at full scale outside of hypothetical neuromorphic settings.

For now, I think we need to rely on probability, statistics and clever functions to drive most approaches forward. "Thinking harder" seems to require unconstrained resources if you need a decision in near real time.

https://arxiv.org/abs/2006.16236 Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention