Evaluating the Reliability and Validity of Practitioners and Consumers’ Advertising Creativity Assessments

Ad creativity, based on the originality, meaningfulness, and craftsmanship of a brand message, is widely studied as an important mechanism of advertising success. These studies are based on practitioners’ and consumers’ creativity assessments. The purpose of this study is to evaluate the creativity assessments of these two groups against empirical standards in terms of intercoder reliability, stability, criterion validity, and predictive validity. To address this aim, sixty Instagram brand messages from the top-100 most interactive Instagram brand accounts were assessed by practitioners, consumers, and trained advertising researchers (for examining criterion validity). The results showed that practitioners and consumers can produce stable creativity assessments provided that the number of coders is large enough. Moreover, the results showed that assessments of both groups differed in terms of criterion validity. Examining the predictive validity showed that both groups yielded different outcome measures. These results (1) confirm that the creativity assessments of practitioners and consumers differ and (2) show that the quality of these assessments are contingent on the group of assessors (practitioners or consumers), sample size, and the assessed creativity dimensions. These findings have important implications for the design, evaluation, and comparability of creativity studies.