Random Permutation Testing Applied to Measurement Invariance Testing with Ordered-Categorical Indicators

We describe and evaluate a random permutation test of measurement invariance with ordered-categorical data. To calculate a p-value for the observed (∆)χ2, an empirical reference distribution is built by repeatedly shuffling the grouping variable, then saving the χ2 from a configural model, or the ∆χ2 between configural and scalar-invariance models, fitted to each permuted dataset. The current gold standard in this context is a robust mean- and variance-adjusted ∆χ2 test proposed by Satorra (2000), which yields inflated Type I errors, particularly when thresholds are asymmetric, unless samples sizes are quite large (Bandalos, 2014; Sass et al., 2014). In a Monte Carlo simulation, we compare permutation to three implementations of Satorra’s robust χ2 across a variety of conditions evaluating configural and scalar invariance. Results suggest permutation can better control Type I error rates while providing comparable power under conditions that the standard robust test yields inflated errors.