International Journal of Technology Assessment in Health Care

METHODS

Setting priorities in clinical and health services research: Properties of an adapted and updated method

Silvina Berraa1, Emília Sáncheza1, Joan M. V. Ponsa2, Cristian Tebéa3, Jordi Alonsoa4 and Marta Aymericha5

a1 Catalan Agency for Health Technology Assessment and Research (CAHTA) and CIBERESP (CIBER en Epidemiología y Salud Pública), Spain

a2 Department of Health, Catalonia; Catalan Agency for Health Technology Assessment and Research (CAHTA); and CIBERESP (CIBER en Epidemiología y Salud Pública), Spain

a3 Catalan Agency for Health Technology Assessment and Research (CAHTA) and CIBERESP (CIBER en Epidemiología y Salud Pública), Spain

a4 IMIM – Hospital del Mar; Pompeu Fabra University Barcelona; and CIBERESP (CIBER en Epidemiología y Salud Pública), Spain

a5 University of Girona

Abstract

Objectives: The objectives of this study is to review the set of criteria of the Institute of Medicine (IOM) for priority-setting in research with addition of new criteria if necessary, and to develop and evaluate the reliability and validity of the final priority score.

Methods: Based on the evaluation of 199 research topics, forty-five experts identified additional criteria for priority-setting, rated their relevance, and ranked and weighted them in a three-round modified Delphi technique. A final priority score was developed and evaluated. Internal consistency, test–retest and inter-rater reliability were assessed. Correlation with experts’ overall qualitative topic ratings were assessed as an approximation to validity.

Results: All seven original IOM criteria were considered relevant and two new criteria were added (“potential for translation into practice”, and “need for knowledge”). Final ranks and relative weights differed from those of the original IOM criteria: “research impact on health outcomes” was considered the most important criterion (4.23), as opposed to “burden of disease” (3.92). Cronbach's alpha (0.75) and test–retest stability (interclass correlation coefficient = 0.66) for the final set of criteria were acceptable. The area under the receiver operating characteristic curve for overall assessment of priority was 0.66.

Conclusions: A reliable instrument for prioritizing topics in clinical and health services research has been developed. Further evaluation of its validity and impact on selecting research topics is required.

Footnotes

We thank the researchers and professionals who participated as reviewers in the 2006 Call for Topics and who voluntarily accepted to participate in this consensus procedure. This made it possible to develop the tool without a need for further funding. The authors also express their gratitude to Cristina Jaldón and Maite Solans for their administrative and technical support, and Drs. Gaietà Permanyer and José Asúa for reviewing previous reports of this work.