Measuring Inter-subjective Agreement on Units and Attributions in Comics with Annotation Experiments

Abstract

The conceptualisation of units of interpretation and analyses remains an inherent issue across comics studies. Despite the many conceptualisations of comics units from numerous theories and disciplines, empirical assessments of their validity as proxies for reader interpretation have yet to receive adequate attention. We argue that unit delineation practically involves classifying groups of visual and textual markings according to type, function or semantic category. Based on this, we present a nascent methodology for collecting and measuring inter-subjective agreement by comics readers on proposed units of comics and their attributes. We create an online tool to facilitate handmade segmentations on digital comic pages and assigning labels or classifications appropriate to the annotation task, resulting in segmentation-attribute pairs. We demonstrate the methodology through two inter-annotator agreement experiments that test a segment-attribute pair of p a n e l  s e g m e n t a t i o n and a judgment of b a c k g r o u n d  l o c a t i o n  i n f o r m a t i o n . The first experiment shows that assigning a binary classification for panel background judgments requires refinements. The second experiment reconceptualises the task to assess agreement on two scalar methods, namely Likert ratings and a continuous scale. We argue that these experiments support the claim that we can build models of structures in comics with an empirical anchor of reader judgments through this methodology.