Assessing metacognitive activities: the in-depth comparison of a task-specific questionnaire with think-aloud protocols

Teaching and assessing metacognitive activities are important educational objectives, and teachers are calling for efficient instruments. The advantages of questionnaires in measuring metacognitive activities are obvious, but serious validity issues appear. For example, correlations of questionnaire data with think-aloud measures are generally moderate to low. An explanation may be that these questionnaires are not constructed in line with the metacognitive activities measured by the think-aloud method. In the present study, a questionnaire is constructed based directly on a taxonomy for coding think-aloud protocols. Twenty ninth-graders studied a text while thinking aloud, after which they immediately received the questionnaire. The overall correlation between the questionnaire and the think-aloud protocols (r = 0.63) was promising. However, scale and item analyses clearly demonstrate some new validity issues. Comparing the questionnaire and the think-aloud results, the students seem to report overt metacognitive activities corresponding more with their behavior reported in the protocols than covert ones. In-depth explorations are presented.