Inter-rater reliability of nursing home quality indicators in the U.S

Vincent Mor, Joseph Angelelli, Richard Jones, Jason Roy, Terry Moore, John Morris

Research output: Contribution to journalArticle

124 Citations (Scopus)

Abstract

Background: In the US, Quality Indicators (QI's) profiling and comparing the performance of hospitals, health plans, nursing homes and physicians are routinely published for consumer review. We report the results of the largest study of inter-rater reliability done on nursing home assessments which generate the data used to derive publicly reported nursing home quality indicators. Methods: We sampled nursing homes in 6 states, selecting up to 30 residents per facility who were observed and assessed by research nurses on 100 clinical assessment elements contained in the Minimum Data Set (MDS) and compared these with the most recent assessment in the record done by facility nurses. Kappa statistics were generated for all data items and derived for 22 QI's over the entire sample and for each facility. Finally, facilities with many QI's with poor Kappa levels were compared to those with many QI's with excellent Kappa levels on selected characteristics. Results: A total of 462 facilities in 6 states were approached and 219 agreed to participate, yielding a response rate of 47.4%. A total of 5758 residents were included in the inter-rater reliability analyses, around 27.5 per facility. Patients resembled the traditional nursing home resident, only 43.9% were continent of urine and only 25.2% were rated as likely to be discharged within the next 30 days. Results of resident level comparative analyses reveal high inter-rater reliability levels (most items >.75). Using the research nurses as the "gold standard", we compared composite quality indicators based on their ratings with those based on facility nurses. All but two QI's have adequate Kappa levels and 4 QI's have average Kappa values in excess of .80. We found that 16% of participating facilities performed poorly (Kappa <.4) on more than 6 of the 22 QI's while 18% of facilities performed well (Kappa >.75) on 12 or more QI's. No facility characteristics were related to reliability of the data on which Qis are based. Conclusion: While a few QI's being used for public reporting have limited reliability as measured in US nursing homes today, the vast majority of QI's are measured reliably across the majority of nursing facilities. Although information about the average facility is reliable, how the public can identify those facilities whose data can be trusted and whose cannot remains a challenge.

Original languageEnglish (US)
Article number1
Pages (from-to)1-13
Number of pages13
JournalBMC health services research
Volume3
DOIs
StatePublished - Nov 4 2003

Fingerprint

Nursing Homes
Nurses
Home Health Nursing
Nursing Assessment
Research
Nursing
Urine
Physicians

All Science Journal Classification (ASJC) codes

  • Health Policy

Cite this

Mor, Vincent ; Angelelli, Joseph ; Jones, Richard ; Roy, Jason ; Moore, Terry ; Morris, John. / Inter-rater reliability of nursing home quality indicators in the U.S. In: BMC health services research. 2003 ; Vol. 3. pp. 1-13.
@article{f22227a900c94114aa656ddd12475495,
title = "Inter-rater reliability of nursing home quality indicators in the U.S",
abstract = "Background: In the US, Quality Indicators (QI's) profiling and comparing the performance of hospitals, health plans, nursing homes and physicians are routinely published for consumer review. We report the results of the largest study of inter-rater reliability done on nursing home assessments which generate the data used to derive publicly reported nursing home quality indicators. Methods: We sampled nursing homes in 6 states, selecting up to 30 residents per facility who were observed and assessed by research nurses on 100 clinical assessment elements contained in the Minimum Data Set (MDS) and compared these with the most recent assessment in the record done by facility nurses. Kappa statistics were generated for all data items and derived for 22 QI's over the entire sample and for each facility. Finally, facilities with many QI's with poor Kappa levels were compared to those with many QI's with excellent Kappa levels on selected characteristics. Results: A total of 462 facilities in 6 states were approached and 219 agreed to participate, yielding a response rate of 47.4{\%}. A total of 5758 residents were included in the inter-rater reliability analyses, around 27.5 per facility. Patients resembled the traditional nursing home resident, only 43.9{\%} were continent of urine and only 25.2{\%} were rated as likely to be discharged within the next 30 days. Results of resident level comparative analyses reveal high inter-rater reliability levels (most items >.75). Using the research nurses as the {"}gold standard{"}, we compared composite quality indicators based on their ratings with those based on facility nurses. All but two QI's have adequate Kappa levels and 4 QI's have average Kappa values in excess of .80. We found that 16{\%} of participating facilities performed poorly (Kappa <.4) on more than 6 of the 22 QI's while 18{\%} of facilities performed well (Kappa >.75) on 12 or more QI's. No facility characteristics were related to reliability of the data on which Qis are based. Conclusion: While a few QI's being used for public reporting have limited reliability as measured in US nursing homes today, the vast majority of QI's are measured reliably across the majority of nursing facilities. Although information about the average facility is reliable, how the public can identify those facilities whose data can be trusted and whose cannot remains a challenge.",
author = "Vincent Mor and Joseph Angelelli and Richard Jones and Jason Roy and Terry Moore and John Morris",
year = "2003",
month = "11",
day = "4",
doi = "10.1186/1472-6963-3-1",
language = "English (US)",
volume = "3",
pages = "1--13",
journal = "BMC Health Services Research",
issn = "1472-6963",
publisher = "BioMed Central",

}

Inter-rater reliability of nursing home quality indicators in the U.S. / Mor, Vincent; Angelelli, Joseph; Jones, Richard; Roy, Jason; Moore, Terry; Morris, John.

In: BMC health services research, Vol. 3, 1, 04.11.2003, p. 1-13.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Inter-rater reliability of nursing home quality indicators in the U.S

AU - Mor, Vincent

AU - Angelelli, Joseph

AU - Jones, Richard

AU - Roy, Jason

AU - Moore, Terry

AU - Morris, John

PY - 2003/11/4

Y1 - 2003/11/4

N2 - Background: In the US, Quality Indicators (QI's) profiling and comparing the performance of hospitals, health plans, nursing homes and physicians are routinely published for consumer review. We report the results of the largest study of inter-rater reliability done on nursing home assessments which generate the data used to derive publicly reported nursing home quality indicators. Methods: We sampled nursing homes in 6 states, selecting up to 30 residents per facility who were observed and assessed by research nurses on 100 clinical assessment elements contained in the Minimum Data Set (MDS) and compared these with the most recent assessment in the record done by facility nurses. Kappa statistics were generated for all data items and derived for 22 QI's over the entire sample and for each facility. Finally, facilities with many QI's with poor Kappa levels were compared to those with many QI's with excellent Kappa levels on selected characteristics. Results: A total of 462 facilities in 6 states were approached and 219 agreed to participate, yielding a response rate of 47.4%. A total of 5758 residents were included in the inter-rater reliability analyses, around 27.5 per facility. Patients resembled the traditional nursing home resident, only 43.9% were continent of urine and only 25.2% were rated as likely to be discharged within the next 30 days. Results of resident level comparative analyses reveal high inter-rater reliability levels (most items >.75). Using the research nurses as the "gold standard", we compared composite quality indicators based on their ratings with those based on facility nurses. All but two QI's have adequate Kappa levels and 4 QI's have average Kappa values in excess of .80. We found that 16% of participating facilities performed poorly (Kappa <.4) on more than 6 of the 22 QI's while 18% of facilities performed well (Kappa >.75) on 12 or more QI's. No facility characteristics were related to reliability of the data on which Qis are based. Conclusion: While a few QI's being used for public reporting have limited reliability as measured in US nursing homes today, the vast majority of QI's are measured reliably across the majority of nursing facilities. Although information about the average facility is reliable, how the public can identify those facilities whose data can be trusted and whose cannot remains a challenge.

AB - Background: In the US, Quality Indicators (QI's) profiling and comparing the performance of hospitals, health plans, nursing homes and physicians are routinely published for consumer review. We report the results of the largest study of inter-rater reliability done on nursing home assessments which generate the data used to derive publicly reported nursing home quality indicators. Methods: We sampled nursing homes in 6 states, selecting up to 30 residents per facility who were observed and assessed by research nurses on 100 clinical assessment elements contained in the Minimum Data Set (MDS) and compared these with the most recent assessment in the record done by facility nurses. Kappa statistics were generated for all data items and derived for 22 QI's over the entire sample and for each facility. Finally, facilities with many QI's with poor Kappa levels were compared to those with many QI's with excellent Kappa levels on selected characteristics. Results: A total of 462 facilities in 6 states were approached and 219 agreed to participate, yielding a response rate of 47.4%. A total of 5758 residents were included in the inter-rater reliability analyses, around 27.5 per facility. Patients resembled the traditional nursing home resident, only 43.9% were continent of urine and only 25.2% were rated as likely to be discharged within the next 30 days. Results of resident level comparative analyses reveal high inter-rater reliability levels (most items >.75). Using the research nurses as the "gold standard", we compared composite quality indicators based on their ratings with those based on facility nurses. All but two QI's have adequate Kappa levels and 4 QI's have average Kappa values in excess of .80. We found that 16% of participating facilities performed poorly (Kappa <.4) on more than 6 of the 22 QI's while 18% of facilities performed well (Kappa >.75) on 12 or more QI's. No facility characteristics were related to reliability of the data on which Qis are based. Conclusion: While a few QI's being used for public reporting have limited reliability as measured in US nursing homes today, the vast majority of QI's are measured reliably across the majority of nursing facilities. Although information about the average facility is reliable, how the public can identify those facilities whose data can be trusted and whose cannot remains a challenge.

UR - http://www.scopus.com/inward/record.url?scp=2942696317&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=2942696317&partnerID=8YFLogxK

U2 - 10.1186/1472-6963-3-1

DO - 10.1186/1472-6963-3-1

M3 - Article

C2 - 14596684

AN - SCOPUS:2942696317

VL - 3

SP - 1

EP - 13

JO - BMC Health Services Research

JF - BMC Health Services Research

SN - 1472-6963

M1 - 1

ER -