Assessing the Understanding of Expressions: A Qualitative Study of Notional-Machine-Based Exam Questions

Joey Bevilacqua

Luca Chiodini

Igor Moreno Santos

Matthias Hauswirth

Background and Context: Notional machines are widespread in computing education. While they often are used to explain programming concepts, prior work on visual program simulation demonstrated their use for assessment. This paper presents a qualitative study of the use of a notional machine identified in prior literature“expression as tree”—as an instrument to assess the understanding of expressions. Objectives: The study answers the following research questions: What are the mistakes that students make when answering questions based on the “expression as tree” notional machine? What underlying reasons plausibly explain these mistakes? Are there plausible relationships between the mistakes and programming language misconceptions? Method: We collect a corpus of 542 hand-drawn expression tree diagrams from 12 exams in 6 university programming courses at two different levels over the course of 4 years. We devise and use a coding approach tailored to the qualitative analysis of those diagrams. Our qualitative data analysis approach is unique due to the specific form of the data—hand–drawn diagrams—that admits a wide variety of mistakes, due to the theoretically well–defined programming language constructs that underpin the visualizations, and due to the fact that for each diagram there exists a unique solution that can serve as a reference. Findings: Our results show that even a single question based on the notional machine is able to reveal a rich variety of mistakes related to the structure, typing, and evaluation of expressions. We identify and categorize 48 mistakes and describe outstanding ones in detail, supported by example diagram snippets. We find mistakes that are plausibly connected to known misconceptions, and others that suggest the existence of new ones. Implications: Our findings shed light on novel pedagogical strategies to teach programming and provide insights instructors can use to transform their practices. The use of the “expression as tree” notional machine as an assessment instrument can provide valuable insights into students’ understanding of several key aspects of programming.