How much do narrative annotations differ? A Proppian case study

The formal study of narratives goes back to the Russian structuralist school, paradigmatically represented by the 1928 study Morphology of the Folktale by Vladimir Propp. Researchers in the field of computational narratology have developed the general Proppian methodology into various formal and computational frameworks for the analysis, automated understanding and generation of narratives. Methodological issues in this research field give rise to concrete research questions such as "How much does the representation of a narrative in a given formal framework depend on subjective decisions of the formalizer?'" touching philosophy of computing and philosophy of information. In order to approach the mentioned question, we consider the process of formal representation of a narrative as a natural analogue of the task of annotation in computational linguistics and corpus linguistics. We use the Russian folktales formalized by Propp and let them be formalized by annotators according to Propp's system, evaluating these results according to the standards of interannotator agreement.