Computerized adaptive assessment of understanding of programming concepts in primary school children

Background and Context. Valid assessment of understanding of programming concepts in primary school children is essential to implement and improve programming education.

Objective. We developed and validated the Computerized Adaptive Programming Concepts Test (CAPCT) with a novel application of Item Response Theory. The CAPCT is a web-based and resource-efficient adaptive assessment of 4489 questions measuring: the understanding of basic sequences, loops, conditions (if & if-else statements), debugging, multiple agents, procedures, and the ability to generalize to a new syntax.

Method. Data was collected through an existing online adaptive practice and monitoring system called Math Garden. We collected 14 million responses from 93,341 Dutch children (ages 4 - 13).

Findings. The CAPCT demonstrated good psychometric qualities because 75% of the variance in question difficulty was explained by differences in item characteristics. The CAPCT demonstrated robustness against adding new participants and adding new items. Differences in player ability (i.e., understanding of CS concepts) were due to differences in age, gender, the number of items played, and prior mathematical ability.

Implications. The CAPCT may be used by teachers to identify the level of programming concept understanding of their pupils, while researchers may use the CAPCT to construct and validate effective teaching resources.