International Journal of Technology Assessment in Health Care


Use of expert knowledge in evaluating costs and benefits of alternative service provisions: A case study

Paul H. Garthwaitea1, James B. Chilcotta2, David J. Jenkinsona3 and Paul Tappendena4

a1 The Open University

a2 The University of Sheffield

a3 University of Aberdeen

a4 The University of Sheffield


Objectives: A treatment pathway model was developed to examine the costs and benefits of the current bowel cancer service in England and to evaluate potential alternatives in service provision. To use the pathway model, various parameters and probability distributions had to be specified. They could not all be determined from empirical evidence and, instead, expert opinion was elicited in the form of statistical quantities that gave the required information. The purpose of this study is to describe the procedures used to quantify expert opinion and note examples of good practice contained in the case study.

Methods: The required information was identified and preparatory discussion with four experts refined the questions they would be asked. In individual elicitation sessions they quantified their opinions, mainly in the form of point and interval estimates for specified variables. New methods have been developed for quantifying expert opinion and these were implemented in specialized software that uses interactive graphics. This software was used to elicit opinion about quantities related to measurable covariates.

Results: Assessments for thirty-four quantities were elicited and available checks supported their validity. Eight points of good practice in eliciting and using expert judgment were evident. Parameters and probability distributions needed for the pathway model were determined from the elicited assessments. Simulation results from the pathway model were used to inform policy on bowel cancer service provision.

Conclusions: The study illustrates that quantifying and using expert judgment can be acceptable in real problems of practical importance. For full benefit to be gained from expert knowledge, elicitation must be conducted carefully and should be reported in detail.


This work forms part of a project to examine “The costs and benefits of bowel cancer services in England,” commissioned by the Policy Programme of the Department of Health. It is also part of a project on the “Elicitation of individuals’ knowledge in probabilistic form,” funded by the Research Methodology Programme of the Department of Health under grant number RM02/JH04/AOH. The work benefited greatly from interaction with other members of the project teams and advisory networks, especially Tony O'Hagan and the experts whose opinions are quantified here. During the work, David Jenkinson was a postgraduate student at the Open University and the work was completed while Paul Garthwaite was a visiting academic at the University of New South Wales.