LilowLa – Resources


PILAR: Corpora for different languages of the Iberian Peninsula

A collection of parallel and/or monolingual corpora from different languages of the Iberian Peninsula. The languages included are:

Corporal for Valencian (dialect of Catalan)

Valencian’s corpora crawled from different Valencian government-owned websites, the Diari Oficial de la Generalitat Valenciana and the Boletín Oficial de la UMH. The repository is divided into directories according to the origin of the corpora. Each directory indicates whether the corpus is parallel or monolingual.

MayanV: Corpora for some Mayan languages

This repository contains a number of parallel corpora between several Mayan languages and Spanish.


Tune ‘n’ distill

This repository contains a pipeline to tune the mBART50 NMT pre-trained model to low-resource language pairs, and then distill the resulting system to obtain lightweight and more sustainable models. The pipeline allows training lightweight models for the translation between English and a specific low-resource language, even if mBART50 has not been pre-trained with the low-resource language.


Scripts and code for training bilingual and multilingual NMT models of Mayan languages.


This repository contains the code needed to run multi-task learning data augmentation (MaTiLDA), a method for data augmentation in neural machine translation.


Webcrawler using Selenium and Firefox that retrieves parallel text from the official Jehova’s Witnesses’ website,, using the sitemap file to visit all available URLs in the languages specified.

Idiomata Cognitor

A multilingual language classifier for several Romance languages. The repository contains an explanation of the languages the classifier is able to recognize, the training details and scripts for using the classifier or training a new one.


Tool implemented in Python that allows to infer if a URL links to a document in a certain language. You can get the most likely language or the probability that a URL links to a document in a given language without access to the content of the linked document.

Parallel URLs Classifier

Tool implemented in Python that allows to infer whether a pair of URLs link to parallel documents (i.e., documents with the same content but written in different languages) without accessing their content. You can either get a textual description positive/negative or the probability that the URL pair links to parallel documents.

Translation models

Coming soon.