Abstract:
The fields of embodied cognition and artificial intelligence are undergoing a renaissance at present and many researchers are exploring human-inspired models of cognition and behaviour. This approach often leads to the creation of large-scale systems models consisting of many interacting components. Such models may be difficult to understand, let alone manipulate or debug. This thesis explores techniques and methods for visualising and interacting with such large-scale neural systems models in real-time. The principal outcome is a suite of user interface tools which are designed and implemented to work with systems models in BL (Brain Language). BL is a multi-purpose modular simulation framework developed at the Laboratory for Animate Technologies. The focus of BL is on implementing neurobehavioural/cognitive models from diverse sources (psychology, physiology, neurology etc.) and linking these models with high-quality 3D computer graphics and audiovisual input/output streams. The ultimate goal is to create intelligent, autonomously animated virtual agents. A key feature of BL is the possibility for almost any variable within the system to be shared with and to drive other variables; outputs from neural network simulations are seamlessly linked to complex 3D animation systems. Modelling and simulation with BL often leads to the large, complex systems models described above. The breadth and generality of BL’s simulation framework requires any user interface tools developed for use with BL to be capable of handling diverse model and data types from a range of sources. The user interface tools implemented are hence similarly general; while they are implemented to work with BL, the techniques used might similarly be applied to other simulation frameworks. This thesis begins with some background information about neurobehavioural/cognitive modelling as well as user interfaces, modular modelling and graph visualisation. This section covers a number of the areas of research most relevant to the user interface tools developed as part of this thesis. BL is itself a novel simulation environment, so this background material also helps to situate BL in a wider scientific context. A more technical introduction to BL follows. While the general concepts of the user interface do not depend upon BL, specific details are often linked to underlying aspects of BL. The interface tools are built using a cross-platform user interface toolkit. Chapter 4 outlines the criteria for choosing a toolkit from the many available options and provides background information about Qt (the selected toolkit). This is followed by an overview of the general user interface developed for use with BL, and discussion of some of the ideas and motivations behind specific design choices. In order to better highlight their utility, the major features of the user interface are introduced in the context of specific models implemented in BL. Each model is introduced before selected user interface features, as well as selected features of BL itself, are highlighted and discussed in detail. Three smaller-scale models are presented first: breathing rhythm generation, the patellar reflex and a model of the basal ganglia. Next, features of BL and of the general user interface for working with larger models are discussed, before a novel model of spike timing-dependent plasticity in corticostriatal synapses is presented. The development and implementation of this model is an additional contribution of this thesis. Lastly, application of the user interface systems (both general and custom) to large-scale, complex systems models is demonstrated. The penultimate chapter introduces BabyX, a flagship project of the Laboratory for Animate Technologies embodying models of cognition, emotion and development in a virtual infant. In conclusion, a prototypical suite of user interface tools has been developed which explores and demonstrates techniques for visualising and interacting with large-scale neural systems models in real-time. These tools have been implemented to work with BL in such a way that multiple user interface systems may be run concurrently with BL’s simulation and 3D visualisation, all in real-time. These interface tools are successfully used by both the author and other members of the Laboratory for Animate Technologies as a means to investigate the dynamic properties of models in BL, aiding in both understanding and model development.