Best-case kappa scores calculated retrospectively from EEG report databases

PURPOSE: The most popular metric for interrater reliability in electroencephalography is the kappa (κ) score. κ calculation is laborious, requiring EEG readers to read the same EEG studies. We introduce a method to determine the best-case κ score (κ_BEST) for measuring interrater reliability between EEG readers, retrospectively. METHODS:: We incorporated 1 year of EEG reports read by four adult EEG readers at our institution. We used SQL queries to determine EEG findings for subsequent analysis. We generated logistic regression models for particular EEG findings, dependent on patient age, location acuity, and EEG reader. We derived a novel measure, the κ_BEST statistic, from the logistic regression coefficients. RESULTS:: Increasing patient age and location acuity were associated with decreased sleep and increased diffuse abnormalities. For certain findings, EEG readers exhibited the dominant influence, manifesting directly as lower between-reader κ_BEST scores for certain EEG findings. Within-reader κ_BEST control scores were higher than between-reader scores, suggesting internal consistency. CONCLUSIONS:: The κ_BEST metric can measure significant interrater reliability differences between any number of EEG readers and reports, retrospectively, and is generalizable to other domains (e.g., pathology or radiology reporting). We suggest using this metric as a guide or starting point for focused quality control efforts.