SKR1: Benchmark for Testing Knowledge About Slovak Realia for Large Language Models
SKR1: Benchmark for Testing Knowledge About Slovak Realia for Large Language Models

Summary/Abstract: Background: To objectively evaluate the capabilities of large language models (LLMs), we need to develop tools that enable such assessment. While numerous benchmarks exist, the vast majority are in English and focus on general knowledge, often overlooking the cultural and factual specifics of smaller countries. Objective: Currently, there is no benchmark that tests LLMs' knowledge of Slovak realia. At the same time, LLM performance in this domain remains inadequate. To objectively measure and compare these capabilities, our goal is to develop and validate a specialized benchmark for assessing LLMs' knowledge of Slovak cultural and factual context. Methods: We created a set of 35 questions on Slovak culture, geography, history and language. We designed them to provide unambiguous answers suitable for automated evaluation. Subsequently, we presented the questions to three major language models —DeepSeek V3, OpenAI GPT-4o and Llama 3. Results: DeepSeek scored 54% of correct answers, OpenAI GPT scored 51% and Llama scored 40%. The models scored best in geography questions. Overall scores show that models are not very good in recognising Slovak realia. Conclusion: We present the benchmark for evaluating large language models on Slovak-related knowledge. Even the most advanced current models, including OpenAI GPT and DeepSeek, answered only around half of the questions correctly. This highlights a significant gap in international LLMs' understanding of culturally specific facts, underscoring the need for specialized, nationally tailored language models.

• Details
• Contents