Probability estimate and the optimal text size Cover Image

Aproksimacija verovatnoća i optimalna veličina jezičkog uzorka
Probability estimate and the optimal text size

Author(s): Aleksandar Kostić, Svetlana Ilić, Petar Milin
Subject(s): Phonetics / Phonology, Lexis, Computational linguistics, South Slavic Languages, Experimental Pschology
Published by: Društvo psihologa Srbije
Keywords: corpus linguistics; reliability of text sample;

Summary/Abstract: Reliable language corpus implies a text sample of size n that provides stable probability distributions of linguistic phenomena. The question is what is the minimal (i.e. the optimal) text size at which probabilities of linguistic phenomena become stable. Specifically, we were interested in probabilities of grammatical forms. We started with an a priori assumption that text size of 1.000.000 words is sufficient to provide stable probability distributions. Text of this size we treated as a "quasi-population". Probability distribution derived from the "quasi-population" was then correlated with probability distribution obtained on a minimal sample size (32 items) for a given linguistic category (e.g. nouns). Correlation coefficient was treated as a measure of similarity between the two probability distributions. The minimal sample was increased by geometrical progression, up to the size where correlation between distribution derived from the quasi-population and the one derived from an increased sample reached its maximum (r=1). Optimal sample size was established for grammatical forms of nouns, adjectives and verbs. General formalism is proposed that allows estimate of an optimal sample size from minimal sample (i.e. 32 items).

  • Issue Year: 41/2008
  • Issue No: 1
  • Page Range: 35-51
  • Page Count: 17
  • Language: Serbian