Evaluation of Python Error Message Interpretation: Study on Students with Different Levels of Programming Experience

Abstract

Error messages are a helpful tool for everyone interested in learning a programming language. Even after learning the language, novice and experienced programmers alike, have to interact with an error message one way or another. However, there is no single programming language that is used
across all platforms and systems, so the programmers have to write programs in many different languages. If the programming languages have similar structures, working with a new language is relatively easy. We have asked the question, whether the same effect could be observed in case of error messages. We designed an online survey which was conducted internationally to measure whether the general experience of programmers from any programming language influences the programmer’s ability to correct python error message. The survey was aimed at students who have some experience with programming. We find in comparison with novices, on average the experienced
programmers find and fix an error in the code given the same
error message with its corresponding code snippet. Additionally,
we see correlations between the average of correctly fixed errors
and number of programming languages that the participants had
experience with, as well as their chosen major and their age.