R-NET - Machine Reading Comprehension with Self-matching Networks

07 Aug 2017

Introduction

R-NET is an end-to-end trained neural network model for machine comprehension.
It starts by matching the question and the given passage (using gated attention based RNN) to obtain question-aware passage representation.
Next, it uses a self-matching attention mechanism to refine the passage representation by matching the passage against itself.
Lastly, it uses pointer networks to determine the position of the answer in the passage.
Link to the paper

Question / Passage Encoder
- Concatenate the word level and character level embeddings for each word and feed into a bidirectional GRU to obtain question and passage representation.
Gated Attention based RNN
- Given question and passage representation, sentence pair representation is generated via soft-alignment of the words in the question and in the passage.
- The newly added gate captures the relation between the question and the current passage word as only some parts of the passage are relevant for answering the given question.
Self Matching Attention
- The passage representation obtained so far would not capture most of the context.
- So the current representation is matched against itself so as to collect evidence from the entire passage and encode the evidence relevant to the current passage word and question.
Output Layer
- Use pointer network (initialized using attention pooling over answer representation) to predict the position of the answer.
- Loss function is the sum of negative log probabilities of start and end positions.
Results
- R-NET is ranked second on SQuAD Leaderboard as of 7th August, 2017 and achieves best-published results on MS-MARCO dataset.
- Using ideas like sentence ranking, using syntax information performing multihop inference and augmenting question dataset (using seqToseq network) do not help in improving the performance.