Bridging the gap between digital humanities and natural language processing: Text cleaning and NLP accuracy evaluation of a sample of 20th century Romanian novels
Bridging the gap between digital humanities and natural language processing: Text cleaning and NLP accuracy evaluation of a sample of 20th century Romanian novels

Summary/Abstract: Although Digital Humanities (DH) and Natural Language Processing (NLP) share common ground, the two academic communities around these disciplines rarely benefit from each other. This study aims to bridge the gap between the two fields of study, by attempting to establish whether off-the-shelf NLP tools are suitable for analysing DH-oriented textual corpora. Our case study focuses on The Digital Museum of the Romanian Novel (DMRN), a digital archive which includes most of the novels in Romanian literature up until 1947. Currently, the archive is composed of more than 1200 digital files in the PDF format, derived from their original print editions. An optical character recognition (OCR) layer has also been added in the process, which opens the possibility of applying NLP techniques to these texts. On the other hand, OCRed texts are notoriously difficult to process, being laden with spelling errors and a significant amount of noise. Our case study had two main objectives: 1) to devise an automated method for cleaning the large quantities of OCRed literary texts found in the archive DMRN, in order to (2) analyse the texts by means of NLP with two of the currently available off-the-shelf models, namely spaCy and Stanza. The sample of our study consists of 15 novels published between 1933 and 1947. The extracted texts have been cleaned by employing a custom Python script and further sampled for NLP tasks such as tokenization, part of speech tagging and lemmatization. We then assessed the accuracy of the results of each separate task through manual validation. Overall, our methodology has proven to be efficient and time-saving with regard to the automatic cleaning process, but also satisfactory in what concerns the accuracy percentages of the performed NLP tasks.

• Details
• Contents