Multimodal Attention in Recurrent Neural Networks for Visual Question Answering

Abstract

Visual Question Answering VQA is a task for evaluating image scene understanding abilities and shortcomings and also measuring machine intelligence in the visual domain Given an image and a natural question about the image the system must ground the question into the image and return an accurate answer in a natural language A lot of progress has been done to address the challenges of this task by combining latest advances in image representation and natural language processing Several recently proposed solutions include attention mechanisms designed to support reasoning These mechanisms allow models to focus on specific part of the input in order to generate the answer and improve its accuracy In this paper we present a novel LSTM architecture for VQA that uses multimodal attention to focus over specific parts of the image and also on specific question words to generate the answer We evaluate our model on the VQA dataset and demonstrate that it performs better than state of the art We also make a qualitative analysis of the results and show the abilities and shortcomings of our model