Improving the validity of quantitative measures in applied linguistics research

In empirical applied linguistics research it is essential that the key variables are operationalized in a valid and reliable way, and that the scores are treated appropriately, allowing for a proper testing of the hypotheses under investigation. The current article addresses several theoretical and practical issues regarding the use of measurement instruments and scores. Starting from an elaborated treatment of validity and validation, it introduces a comprehensive framework for score interpretation and score use. Kane's framework serves as a rich context to raise the right questions about a measurement instruments’ validity, and it provides guidance in addressing questions about validity and score use. Application of the framework is illustrated with examples from a recent second language acquisition study about the effects of recasts (Révész, 2012). The critical generalization inference from the framework, as it applies to measuring productive L2 performance, is then examined in greater detail, discussing the different facets of these kinds of measurements and the way reliability issues are and should be dealt with. The paper concludes with a series of common measurement mistakes in applied linguistics research and recommendations to avoid these. The paper offers a brief checklist for proper quantitative data collection and for adequate data treatment in subsequent analyses.