ML Newsnew | past | best | rss | submit about
Milabot: A Deep Reinforcement Learning Chatbot (arxiv.org)
91 points by indescions_2017 on Sep 9, 2017 | hide | past | pdf | 16 comments on HN

In plain words: A chatbot pools many ways of writing replies and learns from real conversations which one to use for each moment. In head-to-head tests with real users, it beat many rival systems and should keep improving as more chats come in.

Abstract · A Deep Reinforcement Learning Chatbot

We present MILABOT: a deep reinforcement learning chatbot developed by the Montreal Institute for Learning Algorithms (MILA) for the Amazon Alexa Prize competition. MILABOT is capable of conversing with humans on popular small talk topics through both speech and text. The system consists of an ensemble of natural language generation and retrieval models, including template-based models, bag-of-words models, sequence-to-sequence neural network and latent variable neural network models. By applying reinforcement learning to crowdsourced data and real-world user interactions, the system has been trained to select an appropriate response from the models in its ensemble. The system has been evaluated through A/B testing with real-world users, where it performed significantly better than many competing systems. Due to its machine learning architecture, the system is likely to improve with additional data.

Iulian V. Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, Sai Rajeshwar, Alexandre de Brebisson, et al.
arXiv:1709.02349 · cs.CL, cs.AI, cs.LG, cs.NE, stat.ML · submitted Sep 7, 2017 · updated Nov 5, 2017
abstract · pdf · html · 40 pages, 9 figures, 11 tables

add comment on HN

rayuela on Sep 9, 2017 [–]
Really throwing the whole kitchen sink at the problem here. They're basically running a RL algo over an ensemble of every successful NLP algo of the last decade.
reply
cs702 on Sep 9, 2017 | parent [–]
Your summary of the paper is both succint and accurate.

What's really interesting to me is that this approach yields state-of-the-art chatbot results, significantly outperforming other, non-ensemble approaches.

It makes me wonder if the subconscious part of the human mind does something similar, i.e., when faced with the task of articulating a response, does the brain subconsciously evaluate a range of possible responses, each generated by different subsystems, and then choose the best of these responses or some mixture of them, to respond?

reply
PaulHoule on Sep 9, 2017 | parent [–]
Maybe that, or maybe you have a large number of strategies for generating replies, and you can pick a good one ahead of time, so you don't need to do an exhaustive search.

Compare it to how a human chess player looks at several possible moves as opposed to a computer program which will consider thousands.

reply
Hydraulix989 on Sep 10, 2017 | parent [–]
AI is still very much a search problem -- leveraging what computers are actually good at doing to try to mimic doing things that they aren't good at.
reply
otakucode on Sep 9, 2017 [–]
If I ask it a question and then ask it the same question 5 minutes later will I get the same answer? If I tell it something about myself and then ask about it later, will it remember? Even the 'smartest' chat bots I have ever 'spoken' with function like someone with no functioning short-term memory or a totally ephemeral identity that alters with every statement.
reply
hippich on Sep 9, 2017 [–]
Is it implemented somewhere as a real code I can run?
reply
cypher543 on Sep 9, 2017 | parent [–]
A machine learning paper with actual code? Hah! Not likely.
reply
sgt101 on Sep 9, 2017 | parent [–]
Get thee to github, there's a bunch. Not least the whole cannon of Tensorflow, but in terms of research all of the GAN stuff last year was on git as the papers hit.
reply
buildbot on Sep 9, 2017 | parent [–]
Unfortunately a mostly true statement in my experience...

And papers the provide code often do not provide enough or the correct code to replicate the paper. Looking at you, XNOR-Net

reply
thinbeige on Sep 9, 2017 [–]
Slighlty OT: Did run many chatsbots in various fields. Chatbots won't be the next big thing, not at all. Chats are a specific channel for more specific use cases, none of them created any killer app.

Maybe we need more time but chatbots got hyped for too long without any significant traction. Most are web shops/ agencies that build mediocre chatbots for brands nobody is using.

Not bashing chatbots just wondering why this topic gets upvoted again and again.

reply
SpeakMouthWords on Sep 9, 2017 | parent [–]
Amazon Echo and Google Home are chatbots, but with a voice interface. They seem relatively well accepted. Your thoughts?
reply
sgt101 on Sep 9, 2017 | parent [–]
I don't think they are - I think they are narrow assistants - order things, search for things, turn things on and off. They aren't there to have a discussion about your brothers emotional response to the Aphex Twin.
reply
thinbeige on Sep 9, 2017 | parent [–]
Well accepted != next big thing
reply
worldsayshi on Sep 9, 2017 | parent [–]
> just wondering why this topic gets upvoted again and again

Because most chatbots are crap (for real world applications) because chatting is a hard problem but there's a great potential?

reply
novalis78 on Sep 9, 2017 [–]
What's really interesting is how they designed their reward function. At the end of each chat the user can give each conversation a rating. The bot then tries to achieve a higher level of human interaction satisfaction, neat!
reply
ma2rten on Sep 9, 2017 [–]
They say their approach relies on a minimal amount of rules, however the most prominent method in their ensemble is a rule based bot.
reply
About | RSS | RSS (all) | HN arXiv