ML Newsnew | past | best | rss | submit about
A Deep Attention-Based Reinforcement Learning Algorithm for Model-Based Control (arxiv.org)
38 points by ctoth on Mar 28, 2019 | hide | past | pdf | 4 comments on HN

In plain words: The agent learns a model of how its environment changes, then uses attention to focus on the important parts of what it sees when judging how good a state is. This lets it learn good control with fewer real-world trials than trial-and-error-only approaches.

Abstract · VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control

Recent breakthroughs in Go play and strategic games have witnessed the great potential of reinforcement learning in intelligently scheduling in uncertain environment, but some bottlenecks are also encountered when we generalize this paradigm to universal complex tasks. Among them, the low efficiency of data utilization in model-free reinforcement algorithms is of great concern. In contrast, the model-based reinforcement learning algorithms can reveal underlying dynamics in learning environments and seldom suffer the data utilization problem. To address the problem, a model-based reinforcement learning algorithm with attention mechanism embedded is proposed as an extension of World Models in this paper. We learn the environment model through Mixture Density Network Recurrent Network(MDN-RNN) for agents to interact, with combinations of variational auto-encoder(VAE) and attention incorporated in state value estimates during the process of learning policy. In this way, agent can learn optimal policies through less interactions with actual environment, and final experiments demonstrate the effectiveness of our model in control problem.

Xingxing Liang, Qi Wang, Yanghe Feng, Zhong Liu, Jincai Huang
arXiv:1812.09968 · cs.LG, cs.AI, cs.NE · submitted Dec 24, 2018
abstract · pdf

add comment on HN

xiphias2 on Mar 28, 2019 [–]
If you look at the old Singularity Conference presentation of Demis Hassabis (founder of Deep Mind), his main argument for using attention, reinforcement learning and deep convolutional networks (and other known systems that were on his slides) for reaching AGI is to use only techniques that the brain uses to be able to decrease the search space for reaching AGI.

I believe his thinking was already proven, and going for emulating and integrating the known algorithms that the brain uses is the fastest way to reach AGI.

https://youtu.be/Qgd3OK5DZWI

reply
jimfleming on Mar 28, 2019 | parent [–]
> I believe his thinking was already proven, and going for emulating and integrating the known algorithms that the brain uses is the fastest way to reach AGI.

That's a stretch considering AGI has not yet been created by DeepMind or anyone else. Notable is that DeepMind's most prominent successes have relied heavily on MCTS, a classical planning method that doesn't have much relation to neuroscience without a lot of caveats. Their accomplishments on Atari lean a lot more on efficient computing than biological plausibility.

I think the strategy they're actually following (and I believe they've said this more recently) is to use what works and to look to neuroscience when other methods fail. This feels more solid than looking to the brain first to narrow the search space, which is the approach Numenta has taken, and does not scale as easily.

reply
xiphias2 on Mar 28, 2019 | parent [–]
You're right with using MCTS, but ,,Their accomplishments on Atari lean a lot more on efficient computing than biological plausibility'' is a strange thing to write.

Efficient computing is of course needed for AGI, I think that was never a question. The question is what algorithms to use on the computers, and also what computing architectures should be created for those algorithms.

Those Atari simulations were the first ones putting together reinforcement learning and deep convolutional networks AFAIK, and yes, tree search was needed (which human brains are consciously doing, but extremely bad at compared to computers).

Just looking at what works is not enough. There was a strong reason why DeepMind didn't start with modelling language or logical reasoning, like many other people, and the background was based in biology (animal behaviour).

reply
jimfleming on Mar 28, 2019 | parent [–]
My point is that DQN is pretty far removed from the biological equivalent. It's impressive and useful but the main reason it succeeded was not because of some deep insight from neuroscience but because it scaled well (or at least better than alternatives at the time).

EDIT: Richard Sutton (largely credited as the grandfather of RL) has written about this recently: http://incompleteideas.net/IncIdeas/BitterLesson.html

reply
About | RSS | RSS (all) | HN arXiv