Exploring Neuromodulation in General Question Answering

Abstract

There exist problems with a QA model’s ability to generalise to unseen data. Current QA models perform well when trained on a dataset individually, but they often perform worse on other datasets of the same task [105]. In the current QA paradigm, QA models are almost exclusively Transformers; therefore, we aim to improve the Transformer’s generalisation capabilities. In an attempt at such, we introduce the Neuromodulated Transformer (NeMoT): an extension to the Transformer via the entwinement of neuromodulation. We hypothesise that the addition of neuromodulation, when coupled with an environment that encourages it (e.g., multi-task and multi-format learning), will result in better generalisation capabilities. We show that NeMoT showcases better generalisation capabilities than a baseline model of a similar structure and 65 million more parameters; however, further experiments are needed to reach a definitive conclusion. In QA, the ability to comprehend text is essential to the generation of the correct answer. Reading strategies are utilised by humans to help them comprehend text and improve their reading proficiency; they have been integrated into QA models in an attempt to achieve the same effects. As a secondary objective in this thesis, we modify and integrate the answer option interaction reading strategy [109] and high- lighting reading strategy [86] with NeMoT. Results show that incorporating reading strategies with NeMoT improves performance on MQA datasets.