Bilingual Sentence Alignment Using Multilingual Language Models

In this work, we propose a new algorithm for bilingual sentence alignment based on multilingual language models. We apply it to low-resource languages and low-resource biomedical domains of rich languages and compare it with the existing alignment algorithms. We have developed machine translation systems for the WMT20 Biomedical Text translation task for four translation directions and achieved the highest BLEU scores for two of them. We further show the advantage of XLM-R (a completely unsupervised multilingual language model) over LASER (a sentence embedding algorithm that requires some labeled data) for parallel corpus alignment of low-resource languages.