AI in Support of StratCom: The Use and Evaluation of Large Language Models in Less Widely Used Official EU Languages
AI in Support of StratCom: The Use and Evaluation of Large Language Models in Less Widely Used Official EU Languages

Author(s): Eduard Barbu, Somnath Banerjee, Tanya Lim, Liene Zīvere
Contributor(s): Hadley Newman (Editor)
Subject(s): Language studies, Applied Linguistics, Communication studies, Baltic Languages, Security and defense
Published by: NATO Strategic Communications Centre of Excellence
Keywords: Large Language Models (LLMs); Strategic Communication; Narrative Detection; Topic Modelling; Language Processing;
Summary/Abstract: This report presents a systematic evaluation of contemporary large language models (LLMs) on three critical natural language processing (NLP) tasks: narrative detection, topic modelling, and aspect-based sentiment analysis (ABSA) . These tasks are foundational for understanding and interpreting textual data from diverse sources, including news media, official communications, academic publications, and social platforms. They enable the extraction of meaningful insights that inform public discourse, support communication-related strategic decision-making, and inform communication strategies across domains. To ensure methodological rigor and domain relevance, the evaluation frame-work was developed in consultation with strategic communications (StratCom) experts. This framework draws on established practices from both NLP research and StratCom analysis, Consequently, the tasks have been adapted from their conventional academic definitions to better reflect the practical and contextual requirements of real-world applications. The LLMs are evaluated in a zero-shot and few-shot settings on a bilingual dataset comprising documents in both English and Latvian, which enables an assessment of their ability to generalise to domain-specific tasks without additional task-specific fine-tuning. While English benefits from extensive coverage in the training corpora of most models, Latvian poses unique challenges due to its comparatively limited representation in existing datasets1 . This dual-language evaluation assesses the models’ capabilities across both high-resource and low-resource settings, highlighting strengths, limitations, and adaptability in multilingual contexts. To enhance accessibility, the report is organised in a modular structure in which Section 2 introduces the evaluated LLMs, Sections 4,6 and 7 present the narrative detection, topic modelling, and aspect-based sentiment analysis tasks with evaluation results for both English and Latvian, and Section 8 concludes with a summary of key findings and their broader implications for multilingual NLP and StratCom. The code, corpora, and prompt templates used in this evaluation are available in a public GitHub repository. This repository supports reproducibility and further research.

• Details
• Contents