Abstract:
Te reo Maori, New Zealand's only indigenous language, is code-switched with English. Maori speakers are atleast bilingual, and the use of Maori is increasing in New Zealand English. Unfortunately, due to the minimal availability of resources, including digital data, Maori is under-represented in technological advances. Cloud-based multilingual systems such as Google and Microsoft Azure support Maori language detection. However, we provide experimental evidence to show that the accuracy of such systems is low when detecting Maori. Hence, with the support of Maori community, we collect Maori and bilingual data to use natural language processing (NLP) to improve Maori language detection. We train bilingual sub-word embeddings and provide evidence to show that our bilingual embeddings improve overall accuracy compared to the publiclyavailable monolingual embeddings. This improvement has been verified for various NLP tasks using three bilingual databases containing formal transcripts and informal social media data. We also show that BiLSTM with pretrained Maori-English sub-word embeddings outperforms large-scale contextual language models such as BERT on down streaming tasks of detecting Maori language. However, this research uses large models 'as is' for transfer learning, where no further training was done on Maori-English data. The best accuracy of 87% was obtained using BiLSTM with bilingual embeddings to detect Maori-English codeswitching points.
