Don’t Forget the Items: Item-Level Meta-Analytic and Substantive Validity Techniques for Reexamining Scale Validation

Nichelle C. Carpenter, Jooyeon Son, T. Brad Harris, Allison L. Alexander, Margaret T. Horner

Research output: Contribution to journalArticlepeer-review

25 Scopus citations


Researchers are generally advised to provide rigorous item-level construct validity evidence when they develop and introduce a new scale. However, these precise, item-level construct validation efforts are rarely reexamined as the scale is put into use by a wider audience. In the present study, we demonstrate how (a) item-level meta-analysis and (b) substantive validity analysis can be used to comprehensively evaluate construct validity evidence for the items comprising scales. This methodology enables a reexamination of whether critical item-level issues that may have been supported in the initial (often single study) scale validation process—item factor loadings and theorized measurement model fit, as examples—hold up in a larger set of heterogeneous samples. Our demonstration focuses on a commonly used scale of task performance and organizational citizenship behavior, and our findings reveal that several of the items do not perform as may have been suggested in the initial validation effort. In all, our study highlights the need for researchers to incorporate item-level assessments into evaluations of whether construct scales perform as originally promised.

Original languageEnglish (US)
Pages (from-to)616-650
Number of pages35
JournalOrganizational Research Methods
Issue number4
StatePublished - Oct 1 2016
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Decision Sciences(all)
  • Strategy and Management
  • Management of Technology and Innovation


  • construct validation procedures
  • content validity
  • meta-analysis
  • quantitative research
  • reliability and validity


Dive into the research topics of 'Don’t Forget the Items: Item-Level Meta-Analytic and Substantive Validity Techniques for Reexamining Scale Validation'. Together they form a unique fingerprint.

Cite this