Assessing the Understanding of Expressions: A Qualitative Study of Notional-Machine-Based Exam QuestionsJoey BevilacquaLuca ChiodiniIgor Moreno SantosMatthias Hauswirth
Joey Bevilacqua
Luca Chiodini
Igor Moreno Santos
Matthias Hauswirth
2024
Proceedings of the 24th Koli Calling International Conference on Computing Education Research
Background and Context:
Notional machines are widespread in computing education. While they often are used to explain programming concepts, prior work on visual program simulation demonstrated their use for assessment. This paper presents a qualitative study of the use of a notional machine identified in prior literature“expression as tree”—as an instrument to assess the understanding of expressions.
Objectives:
The study answers the following research questions: What are the mistakes that students make when answering questions based on the “expression as tree” notional machine? What underlying reasons plausibly explain these mistakes? Are there plausible relationships between the mistakes and programming language misconceptions?
Method:
We collect a corpus of 542 hand-drawn expression tree diagrams from 12 exams in 6 university programming courses at two different levels over the course of 4 years. We devise and use a coding approach tailored to the qualitative analysis of those diagrams. Our qualitative data analysis approach is unique due to the specific form of the data—hand–drawn diagrams—that admits a wide variety of mistakes, due to the theoretically well–defined programming language constructs that underpin the visualizations, and due to the fact that for each diagram there exists a unique solution that can serve as a reference.
Findings:
Our results show that even a single question based on the notional machine is able to reveal a rich variety of mistakes related to the structure, typing, and evaluation of expressions. We identify and categorize 48 mistakes and describe outstanding ones in detail, supported by example diagram snippets. We find mistakes that are plausibly connected to known misconceptions, and others that suggest the existence of new ones.
Implications:
Our findings shed light on novel pedagogical strategies to teach programming and provide insights instructors can use to transform their practices. The use of the “expression as tree” notional machine as an assessment instrument can provide valuable insights into students’ understanding of several key aspects of programming.
Study InformationManually extracted from the paper by the Progmiscon.org team
Programming Languages
Java
Method
Qualitative and quantitative systematic research
Subjects
542 hand-drawn expression trees from 12 exams in 6 university courses
Phenomena Studied
Diagram mistakes
Mistakes in student-drawn expression trees, related to structure, types, or valuesArtifact
https://zenodo.org/records/13898595Related MisconceptionsReferenced by 24 Misconceptions
ArithmeticPlusPrecedes — Java
Addition has higher precedence than string concatenationArrayAllocationWithoutNew — Java
Arrays are created without the new keywordArrayHasLengthMethod — Java
To get the length of an array, one needs to call its length methodCallRequiresVariable — Java
One needs a variable to invoke a methodConstructorWithoutNew — Java
One can write the constructor name, without new, to instantiate a classElseAlwaysExecutes — Java
The else branch of an if-else statement always executesInlineCallInExpressionTree — Java
The expression tree of an expression involving a call inlines the call's computation of the returned valueInlineVariableInExpressionTree — Java
The expression tree of an expression involving a variable inlines the variable's definitionLiteralNoExpression — Java
A literal is not an expressionLiteralString — Java
When passing a literal string as argument to a method, no quotes are neededNoAtomicExpression — Java
Expressions must consist of more than one pieceNoAtomicExpression — JavaScript
Expressions must consist of more than one pieceNoAtomicExpression — Python
Expressions must consist of more than one pieceObjectsMustBeNamed — Java
A variable is needed to instantiate an objectObjectsMustBeNamed — Python
A variable is needed to instantiate an objectParenthesesOnlyIfArgument — Java
() are optional for method calls without argumentsParenthesesOnlyIfArgument — Python
() are optional for function calls without argumentsRightToLeftChaining — Java
Chained accesses are invoked from right to leftRightToLeftChaining — Python
Chained accesses are invoked from right to leftSingleQuoteString — Java
String literals can be in single quotesTargetTyping — Java
The type of a numerical expression depends on the type expected by the surrounding contextThisChildInCallExpression — Java
Instance method or constructor call nodes in expression trees have a child labeled "this"VariablesHoldExpressions — Java
= stores an expression in a variableVariablesHoldExpressions — Python
= stores an expression: it stores a reference to the expression in a variable