about
Are You Talking to a Machine? Image question answering with neural networks (arxiv.org)
4 points by zan2434 on May 22, 2015 | hide | past | pdf | discuss on HN

In plain words: A system reads questions and images, then writes an answer in a word, phrase, or sentence, trained on 150,000 images with Chinese questions and English translations. When judges mixed human and machine answers, they could not tell them apart in 64.7% of cases.

Abstract · Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering

In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: http://idl.baidu.com/FM-IQA.html

Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, Wei Xu
arXiv:1505.05612 · cs.CV, cs.CL, cs.LG · submitted May 21, 2015 · updated Nov 2, 2015
abstract · pdf · html · Dataset released on the project page, see http://idl.baidu.com/FM-IQA.html ; NIPS 2015 camera ready version

add comment on HN