about
Vision-LSTM: xLSTM as Generic Vision Backbone (arxiv.org)
5 points by tosh on Jun 7, 2024 | hide | past | pdf | discuss on HN

In plain words: An updated LSTM that gates information more sharply and stores memories in parallel becomes a vision backbone reading image patches top to bottom, then bottom to top on alternating layers. Tests show it works promisingly as a general-purpose backbone, suggesting LSTM-style blocks could rival transformers.

Abstract

Transformers are widely used as generic backbones in computer vision, despite initially introduced for natural language processing. Recently, the Long Short-Term Memory (LSTM) has been extended to a scalable and performant architecture - the xLSTM - which overcomes long-standing LSTM limitations via exponential gating and parallelizable matrix memory structure. In this report, we introduce Vision-LSTM (ViL), an adaption of the xLSTM building blocks to computer vision. ViL comprises a stack of xLSTM blocks where odd blocks process the sequence of patch tokens from top to bottom while even blocks go from bottom to top. Experiments show that ViL holds promise to be further deployed as new generic backbone for computer vision architectures.

Benedikt Alkin, Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, Johannes Brandstetter
arXiv:2406.04303 · cs.CV, cs.AI, cs.LG · submitted Jun 6, 2024 · updated Feb 20, 2025
abstract · pdf · html · Published as a conference paper at ICLR 2025, Github: https://github.com/NX-AI/vision-lstm

add comment on HN