Ordered or scrambled: how the forward sequencing of multiple choice questions affects test item scores

We analyse the impact on student performance of scrambling the order of multiple choice questions (MCQs) in examinations. We test if students perform better on a question if it is preceded by a question that relates to either the same or the preceding lecture in the course (forward sequenced). Previous research has been unable to find clear evidence of a forward sequencing effect, which we argue could be related to their focus on test level outcomes. By considering test-item level results instead, using a sample of 16,127 MCQ scores from 12 Microeconomics examinations at Radboud University, the Netherlands, we find strong evidence that the forward sequencing of MCQs on examinations positively impacts student performance at the test-item level. An average student has a higher likelihood of answering correctly a MCQ that is in sequence than one that is not. This result holds after controlling for student ability, question difficulty, and a range of other determinants of student performance. The positive effect of forward sequencing builds up in case of a streak of sequenced MCQs. We conclude that the scrambling of MCQs on examinations is not innocuous and we call for consistent scrambling practices across examination versions.