Niektoré problémy pri tvorbe ukrajinského webového korpusu Araneum Ucrainicum
Some challenges in creating the Ukrainian web corpus Araneum Ucrainicum
Author(s): Vladimír BenkoSubject(s): Language studies, Electronic information storage and retrieval, Applied Linguistics, Computational linguistics, Eastern Slavic Languages, ICT Information and Communications Technologies
Published by: AV ČR - Akademie věd České republiky - Ústav pro jazyk český
Keywords: Aranea web-crawled corpora; deduplication; ensemble method of lemmatization and tagging; NoSketch Engine corpus manager; tokenization; Ukrainian language
Summary/Abstract: The Ukrainian corpus is being developed in the framework of the Aranea project, aiming to create web-crawled corpora for several (primarily) European languages. The data were downloaded from the Internet across multiple sessions between 2014 and 2026, so the texts also cover the discourse of the pandemic period and the Russo-Ukrainian war. In this paper, we describe the main specific challenges and language-motivated decisions related to the creation of this Ukrainian corpus; particularly the filtering, deduplication, and tokenization of the texts. We present the ensemble method for lemmatization and morphosyntactic annotation using four different tools, followed by aggregation, classification of individual corpus tokens using regular expressions, and the identification of the language of individual corpus sentences. Using a sample of 170 million tokens from the latest data download session, we demonstrate searching the corpus by means of the NoSketch Engine corpus manager and the calculation of subcorpus keywords using the A. Kilgarriffʼs method.
Journal: Naše řeč
- Issue Year: 109/2026
- Issue No: 2
- Page Range: 93-123
- Page Count: 31
- Language: Slovak
