Hostname: page-component-7c8c6479df-995ml Total loading time: 0 Render date: 2024-03-29T06:27:39.443Z Has data issue: false hasContentIssue false

Eliciting expert opinion on the effectiveness and practicality of interventions in the farm and rural environment to reduce human exposure to Escherichia coli O157

Published online by Cambridge University Press:  07 July 2011

P. CROSS*
Affiliation:
School of the Environment, Natural Resources and Geography, College of Natural Sciences, Bangor University, Gwynedd, UK
D. RIGBY
Affiliation:
Economics, School of Social Sciences, University of Manchester, UK
G. EDWARDS-JONES
Affiliation:
School of the Environment, Natural Resources and Geography, College of Natural Sciences, Bangor University, Gwynedd, UK
*
*Author for correspondence: Dr P. Cross, School of the Environment, Natural Resources and Geography, College of Natural Sciences, Bangor University, Gwynedd LL57 2UW, UK. (Email: afs202@bangor.ac.uk)
Rights & Permissions [Opens in a new window]

Summary

Few hard data are available on emergent diseases. However, the need to mitigate and manage emergent diseases has prompted the use of various expert consultation and opinion elicitation methods. We adapted best-worst scaling (BWS) to elicit experts' assessment of the relative practicality and effectiveness of measures to reduce human exposure to E. coli O157. Cattle vaccination was considered the most effective and hand-washing was considered the most practical measure. BWS proved a powerful tool for expert elicitation as it breaks down a cognitively burdensome process into simple, repeated, tasks. In addition, statistical analysis of the resulting data provides a scaled set of scores for the measures, rather than just a ranking. The use of two criteria (practicality and effectiveness) within the BWS process allows the identification of subsets of measures judged as potentially performing well on both criteria, and conversely those judged to be neither effective nor practical.

Type
Original Papers
Copyright
Copyright © Cambridge University Press 2011

INTRODUCTION

Newly emergent pathogens in the environment pose important challenges to public health policy. Since 1980, 87 new human pathogen species have been discovered, many of which are associated with animal reservoirs [Reference Woolhouse and Gaunt1, Reference Morgan2] such as Campylobacter, Escherichia coli O157 and Yersinia [Reference Tauxe3]. Emerging pathogen risks are, by definition, associated with incomplete evidence bases and there is a recurring demand for the scientific community to inform policy-makers about risk management even though systematic evidence is often limited [Reference Henson4].

In the absence of a systematic evidence base, alternative approaches for the management of livestock diseases have been sought. These approaches seek to assist the management of uncertainty [Reference Henson4Reference van Schaik6]. Often this involves making recommendations on the best available data, while acknowledging the uncertainty involved in the evidence and hence in the resulting recommendations. Expert opinion is a source of information often sought in this context, and identifying expert consensus regarding risks and the appropriate means of their management represents an attractive option in many situations. However, the appropriate means by which experts are identified and the process by which their opinions are elicited remains contested. In addition, even if the views of the correct experts for consultation are identified, there may be no, or only partial, consensus among them. This is not surprising as the extent of consensus within a scientific community may be related to the level of uncertainty regarding the issue at hand and such uncertainty is endemic to the management of emergent pathogen risks.

A number of methods have been used to elicit expert opinion regarding pathogenic risk. Some studies have employed standard questionnaire approaches, for example, asking expert panels to score a range of measures using Likert scales or similar response formats [Reference Sørensen7, Reference Garabed8]. Other studies have asked an expert panel to weight the relative importance of risk factors [Reference Fish5, Reference Garabed8]. A number of studies have also employed market research techniques such as conjoint analysis [Reference van Schaik6, Reference Valeeva9] in which respondents complete structured survey tasks (making choices, ranking, etc.), generated via an experimental design, in order to explore trade-offs between candidate measures as a means of prioritizing them (see [Reference van Schaik6, Reference Cross, Williams and Edwards-Jones10, Reference Valeeva11]). Conjoint analysis originates in market research where it is used to derive estimates of individuals' preferences for a product or service, and/or its component characteristics [Reference Orme12]. There are various forms of conjoint analysis [adaptive conjoint, choice-based conjoint (CBC), etc.] but they have in common the understanding that a product comprises a number of attributes the importance of which differs across consumers [Reference Mennecke13].

The application of these techniques to disease management has involved, inter alia, identifying the relative effectiveness of potential management measures and how they might be bundled into effective interventions. Cross et al. [Reference Cross, Williams and Edwards-Jones10], for example, use adaptive conjoint analysis (ACA) to explore the effectiveness of trade-offs between interventions proposed for the control of bluetongue. The ACA approach used by Cross and colleagues is well suited to situations with a small number of control measures, often differing in their levels, which would typically be bundled together by practitioners. It is less suitable when there is a very large set of stand-alone control measures under consideration. In this situation a form of conjoint analysis, termed best-worst scaling (BWS) may offer more potential [Reference Coltman, Devinney and Keating14].

We apply the technique to investigate the management of E. coli O157 in agriculture and the wider rural environment. E. coli O157 was reported as a human pathogen in 1982 [Reference Riley15] and a number of outbreaks have subsequently occurred, typically associated with foodborne infection [Reference Bell16, Reference Kassenborg17]. More recent outbreaks have shown environmental contamination to be an increasingly common pathogen transmission pathway [Reference Locking18Reference Parry20]. These outbreaks, alongside evidence on shedding levels, persistence in the environment [Reference Ogden21, Reference Ogden, MacRae and Strachan22] and re-colonization [Reference Khanna23] have highlighted the need to focus on environmental aspects of mitigation [Reference Solecki24].

Given increased calls in the UK for action on the management of E. coli O157 risk, there is a need to evaluate potential management measures in farm and rural settings. This need for action has become more urgent in light of recent outbreaks such as that at Godstone farm in Surrey in 2009 when 93 people became infected, with 17 developing complications and eight requiring kidney dialysis [Reference Griffin25]. Hence there is an imperative to understand, and potentially reconcile, expert opinion on the potential of a range of interventions to reduce the risk of E. coli O157 exposure to humans in an environmental setting.

This study investigates control measures applicable in the farm and rural environment. The scope of the farm control measures was limited to pre-lairage for livestock and the farm gate for vegetables. The orientation on the farm and rural environment is one of the novel aspects of the research since many other studies [Reference Lynch, Tauxe and Hedberg26, Reference Newell27] have focused on risks of exposure and associated control measures occurring later in the food chain, often concerning food processing, storage and handling rather than direct environmental exposure. The considered measures affect both direct environmental human exposure (e.g. via camping, petting farms, etc.) and exposure via food (reducing shedding levels of livestock entering the food chain). The focus on the farm and rural environment served to both address a knowledge gap and to keep the range of control measures considered manageable.

This paper describes and critically reflects upon the use of a novel technique (BWS) to elicit expert opinion regarding the ‘effectiveness’ and ‘practicality’ of measures to manage E. coli O157 risk in the farm and rural environment. It has a methodological focus in terms of outlining and testing a novel approach to expert elicitation. It has a substantive focus also, concerning the relative practicality and effectiveness of O157 control measures, and the potential complementarities and conflicts regarding their performance using these two criteria.

METHODOLOGY

BWS

BWS is a choice-based technique that requires respondents to make repeated choices between sets of options [Reference Auger, Devinney and Louviere28]. We first explain the method in general before indicating how it is applied to pathogen management in this study.

In a BWS study respondents are presented with repeated, varying, sets of (typically four or five) items and asked to identify at the extremes of their preferences. For example in a set of four control measures (A–D), the respondent indicates which they consider to be the ‘most effective’ and ‘least effective’ measures. If A is more effective than D then information is obtained for five of the six possible pairs of measures within the set. It is known that A is viewed as more effective than B, C and D (three pairs) and that B and C are more effective than D (two pairs). The only pair on which no information is available is B–C. Similarly, for a 5-item set we gain information on 18/25 paired combinations. The process is repeated with new sets (generated via an experimental design) and in each case choices are made at the extreme using whatever criteria is specified by the researcher. The data are analysed by counting procedures or by estimating choice models to derive importance weights across the items featured in the sets.

The BWS method is typically used when information is sought over a large set of items [Reference Marley and Louviere29]. It holds a number of advantages over other rating and ranking techniques [Reference Lusk and Briggeman30]. First, approaches that use some form of scaling (e.g. scoring an item on a scale of 1–5) assume that the distance properties on the scale are equal. Where scaling properties are unknown the possibility of transforming these into parametric data is not reliable [Reference Jaeger31]. Second, scales can be interpreted differently between individuals. For instance, one respondent may score an item as a ‘4’ while another scores it a ‘5’ when both respondents rate the item equally but have different conceptions of the value of the scores on the scale. Third, respondents are not necessarily compelled to choose one option over another, and frequently they score all of the items with equal importance [Reference Lee, Soutar and Louviere32]. This relates to another problem common to rating/ranking approaches, namely the lack of discriminatory power between items [Reference Jaeger31]. Finally, interpretation of the scores is comparatively simple as the number of times an item is selected as best, minus the number of times an item is selected as worst, approximates the true scale value [Reference Auger, Devinney and Louviere28, Reference Marley and Louviere29].

In this study we use the approach in a novel manner. We define the sets in terms of E. coli O157 control measures and use two choice criteria with those sets: effectiveness (most/least effective) and, in a second stage, practicality (most/least practical to implement). The resulting data are then analysed and the perceptions of the interventions' performance in terms of both practicality and effectiveness assessed. We now turn to the research process before setting out the analytical models employed and the results derived from them.

Identification of O157 control measures and expert sample

Intervention inclusion

Measures to reduce human exposure to E. coli O157 relating to the farm and rural environment were identified from peer-reviewed papers and grey literature. The scope of the farming system considered was pre-lairage for livestock and the farm gate for vegetables. The initial list comprised 99 interventions which was deemed excessive for a cognitively bearable BWS exercise and hence an initial expert consultation was undertaken to reduce the set to a more manageable size.

Expert group recruitment

The composition of the expert panel was contingent upon the scope of the interventions selected for evaluation. The panel reflected the study's focus on interventions applicable in the rural and farm environment.

Experts were identified via authorship of relevant publications and from peer networks. A snowball technique was adopted whereby invited experts were also encouraged to suggest other members in their field of expertise who might be willing to participate in the study. Such an approach taps into pre-existing professional expert networks and allows the researchers to distance themselves from the expert selection process [Reference Garabed8]. Follow-up contact was made by phone and e-mail to encourage participation. Experts were invited from several different disciplines and academic sectors including public health; veterinary science; food microbiology; agricultural/environmental microbiology; clinical microbiology; land management and pathogen management; risk assessment; communicable disease epidemiology; molecular ecology (details of the expert sample can be seen in Table 1).

Table 1. Categories of respondent expertise invited to participate and their response rates

Expert elicitation round 1: shortlisting

The first round of expert consultation required experts to decide which of the 99 interventions should be retained for further evaluation based upon their effectiveness in reducing human exposure to E. coli O157. Experts classified each intervention as ‘priority retain’, ‘retain’, ‘don't retain’ and ‘don't know’. These votes were then scored with values of ‘2’, ‘1’, ‘–1’ and ‘0’ respectively (the full set of interventions is available upon request).

Expert elicitation round 2: BWS

In the second round respondents were asked to re-evaluate the shortlist of measures from round 1 using the BWS methodology. This involved respondents assessing 12 sets, each containing five measures, with the combinations of measures within sets determined by an experimental design. An example of a BWS set is shown in Table 2. In the first stage respondents indicated the ‘most’ and ‘least’ effective measures in each set they faced. The process was then repeated with the ‘most’ and ‘least’ practical measure chosen in each set.

Table 2. Example of a ‘practicality’ best-worst scaling choice set

Analysis

The BWS data on perceptions of the measures' practicality and effectiveness can be analysed in a number of ways. The first form of analysis involves counting rather than estimation. The analyst calculates on what proportion of occasions it was shown each measure was selected as ‘most’ and ‘least’. A more sophisticated analysis involves estimation of practicality and effectiveness scores via a choice model based on random utility (RU) theory [Reference McFadden33] which dominates the empirical analysis of choice in many fields. We briefly explain the approach below.

Faced with a BWS set the respondent is asked to indicate the best and worst performing measure, hence the respondent is choosing the two measures with the maximum difference in performance between them. In a set of K measures there are K(K−1) best-worst combinations. The objective is to retrieve estimates of the sample's performance scores that best explain the observed pattern of best-worst choices, and it is the RU choice model which permits this. While such choice models typically involve people choosing a single, most preferred option in this study we consider scales defined in terms of practicality, and then effectiveness. We consider there to be scale of practicality on which measures can be located and refer to φA as the position of measure A on that scale. Respondent n's unobserved practicality score for measure A is given by

(1)

where ∊nA is an error term, the inclusion of which creates a probabilistic rather than deterministic choice model. The probability of person n choosing any pair of best-worst choices, for example measures A and D, is given by the probability that (P nAP nD) exceeds all other K(K−1) performance differences within the BWS set.

The model is statistically implementable via the assumption that the error term, ∊nA, has an extreme value type I (Gumbel) distribution. This means that the probability that A and D are chosen as most and least practical, respectively, is given by the standard conditional logit formulation:

(2)

Maximum-likelihood estimation of formula (2) involves retrieval of estimates of the φ performance scores which maximize the likelihood of the observed pattern of best-worst choices being observed. The approach is relative, i.e. the estimated practicality and effectiveness scores are relative to each other on an arbitrary scale, with one measure's performance score normalized at zero for identification purposes.

The conditional logit model in formula (2) does not, as specified, include personal characteristics and as such does not allow for the investigation of heterogeneity among the sample. One could extend the model to include characteristics, for example we could allow, and test whether, specific performance scores differ across respondents of differing expertise or demographic profiles. An alternative approach to the accommodation of heterogeneity is the infinite mixture, or mixed logit, model [Reference McFadden and Train34, Reference Train35]. In this model the importance scores are assumed to be drawn from a distribution the mean and standard deviation of which are estimated. An attraction of this model is that as well as identifying a point estimate of the mean importance score, estimates can be derived for each survey respondent, conditional on the best-worst choice data and the estimated population parameters. More formally, person n's performance score for measure A (φnA) is drawn from a distribution with mean (φ*A) and standard deviation σA. Person n's performance score deviates from the sample mean (φ*A) via a disturbance term, ∇, where ∇~N(0, 1):

(3)

The derivation of respondent-specific performance scores is particularly attractive in this study given the desire to examine the degree of consensus or disagreement among the expert sample on the performance of the O157 control measures. We refer readers to [Reference McFadden and Train34, Reference Train35] for more on the estimation of the mixed logit model, noting only that we estimate the model using Bayesian methods [Reference Rigby, Balcombe and Burton36] with the sampler run for 20 000 iterations (the ‘burn in’) before the parameters were recorded, followed by another 20 000 iterations with which to summarize the posterior distribution of the measures' performance scores.

RESULTS

Expert elicitation round 1

For the first round, 53 experts were contacted and 31 completed the survey giving a response rate of 58·5%. In round 2 the snowball sampling process meant 70 experts were invited to complete the survey, 41 of whom did so, giving a response rate of 60%. Thirty-five of the respondents in round 2 gave details of their expertise and these are described in Table 1. Response rates were higher for agricultural/environmental microbiologists (75%) compared to public health experts (48%) which possibly reflects a greater level of confidence for the latter group ranking interventions primarily located in the rural and agricultural environment.

The resultant rankings and the associated suggestion that the top 30 measures would be taken forward to round 2 for more detailed assessment of their practicality and effectiveness were circulated (full details of the results from this round are available upon request). Respondents were invited to appeal against the inclusion/exclusion of any measures but no such appeals were forthcoming. As a result the top 30 measures (Table 3) were included in round 2 which used BWS.

Table 3. List of top 30 interventions selected in round 1 for further appraisal in round 2

HACCP, Hazard Analysis Critical Control Point; COSHH, control of substances hazardous to health.

BWS

Within both the practicality and effectiveness stages of the BWS survey 41 respondents completed a total of 492 BWS sets, making two choices within each one. We first considered the choice data descriptively rather than via model estimates. These estimates provide a powerful descriptive of the typical assessment of a measure and also the degree of consensus. For example measure 27 (reducing stocking densities by 50%) was ranked least practical in almost 50% of the sets in which it appeared and most practical in less than 5% of the sets. In contrast measure 19 (cattle vaccination) was ranked least practical in 35% of the sets but also most practical in 30% of the sets. Cattle vaccination was voted the most effective measure in 72% of the sets in whch it appeared and was rarely chosen as ‘least effective’.

Assessment of the measures' relative practicality and effectiveness was also investigated via estimation of the choice model described in the Analysis section. Table 4 reports results from this model estimated (separately) on both the practicality and effectiveness BWS data. The results in Table 4 show the sample mean point estimates of the practicality and effectiveness scores for all 30 measures. These estimates have been rescaled to sum to 100 to aid interpretation and comparison across the practicality and effectiveness models. In addition to the point estimates, 95% confidence intervals are provided for each parameter estimate.

Table 4. Sample mean point estimates of the effectiveness and practicality scores for all 30 measures

Cattle vaccination, the removal of animals from the proximity of private water supplies (PWS); the removal of high shedding animals prior to slaughter and the monitoring (and where appropriate treatment) of PWS; were estimated to be the most effective measures. The results indicate that the annual inspections of septic tanks, the daily cleaning of in-house water troughs and the use of double fencing to prevent contact between (sub)herds were, on average, regarded as least effective.

In terms of practicality, encouraging farmers and farm visitors to wash hands following contact with farm animals, keeping livestock away from stores of ready-to-eat crops and a 4-week grazing ban following the spreading of animal waste were seen as the most practical measures. Halving stocking densities, banning manure spreading within 500 m of ready-to-eat crops and using vegetative buffer strips to control contamination of ready-to-eat crops from run-off were seen as the least practical to implement.

The mean practicality and effectiveness scores of the measures are combined in Figure 1. The estimated scores are zero-centred and plotted in practicality and effectiveness space in which the axes represent the mean (zero) practicality and effectiveness scores within the sample. Hence measures plotted above 0 on the y-axis have higher than average effectiveness scores, those with scores over 0 on the x-axis have above average practicality scores. Those measures located in the upper right quadrant are regarded as performing relatively well in terms of practicality and effectiveness, those in the bottom left quadrant perform relatively poorly on both criteria. Measures located in the remaining two quadrants highlight the trade-off between the practicality and effectiveness of some measures. Those interventions located above in the upper right-hand segment of the 2×2 plot might be considered candidate interventions for consideration by policy-makers as they score positively for effectiveness and/or practicality.

Fig. 1. Zero-centred scatterplot of mean effectiveness and practicality scores for the 30 control measures.

As indicated in the Analysis subsection of the Methodology section, the choice model allows the estimation of respondent-specific scores as well as the estimate of the mean score for the sample as a whole. Analysis of these scores for a single measure allows a consideration of the degree of consistency or disagreement among the sample regarding the measure's performance. This is highlighted in Figure 2 which shows the spread of practicality and effectiveness scores for two example measures. There was a high degree of agreement regarding the effectiveness of vaccination to reduce pathogen loads, but opinion appeared to be divided regarding the practicality of its implementation. For some measures there was little consistency regarding either criteria, leading to a more uniform distribution of scores in practicality and effectiveness space as shown for intervention 16 [Hazard Analysis Critical Control Point (HACCP) requirement for manure spreading].

Fig. 2. Respondent-specific practicality and effectiveness scores for two control measures (each symbol denotes the practicality and effectiveness scores of a single expert).

DISCUSSION

This study has proposed and implemented BWS as an expert opinion elicitation tool. BWS has been used to evaluate the practicality and effectiveness of measures to reduce human exposure to E. coli O157 in rural and farm environments. Following an initial shortlisting process 30 measures were included in the BWS survey.

The BWS process, completed online, allowed the involvement of many experts from a diverse range of expertises (public health, environmental microbiology, epidemiology, etc.) in varied, often geographically remote, locations. Asking those experts to simply rank the full set of 30 measures would have been extremely cognitively demanding, hence the use of BWS with the repeated ranking over smaller subsets. The study achieved a high response rate with low levels of dropout mid-BWS process, which we regard as evidence that the process was cognitively bearable and intuitive.

The ex-post analysis employed counts and the estimation of a choice model on the best-worst data which allowed a full, scaled ranking to be derived. The model results include both sample-mean, and respondent-specific, estimates of the measures' practicality and effectiveness. Analysis of the individuals' scores for a single measure allows the degree of consensus or disagreement to be investigated.

We regard the BWS as highly flexible and suitable when evaluation needs to be multi-dimensional. In this case two facets of performance were considered (practicality and effectiveness). This allows the location of each measure in a multi-dimensional performance space (Fig. 1). This 2×2 plot is easily interpretable and provides a powerful stimulus for further discussion, as has been evident from our use of it at, inter alia, disease management workshops and meetings of the Advisory Committee on the Microbiological Safety of Food and meat industry bodies.

As to the substantive results, hand washing was regarded as by far the most practical measure. The separation, in time or space, of livestock waste from water supplies, water courses, ready-to-eat crops and the general public were common to many of the measures seen as most practical and effective. In addition some more technical ‘fixes’ were identified as most effective: concerning livestock and PWS. The three measures viewed as most effective were vaccination, removal of high shedding animals, and monitoring and treatment of PWS.

Some measures which recent research [Reference Ellis-Iversen37] has suggested as promisingly effective in this context were regarded as less effective (requiring in-house water troughs to be cleaned daily, maintaining dry livestock bedding). This raises the issue of managing the tension between a common expert opinion and fragments of research evidence which only some may be party to, or accept. The issue of bedding management was highlighted by the Griffin Report into the Godstone outbreak. Another notable feature of our results is that the prevention of children under the age of 11, and other vulnerable groups, from coming into contact with animals at petting, or public visitor farms was regarded as both highly effective and practical.

We note that the Godstone outbreak took place between the first and second rounds of our elicitation process which may have affected the perceived performance of both this measure and hand washing. It is evident that an empirical microbiological evaluation of the exposure hazard would remain unaffected by such an outbreak and underlines the importance of basing policy decisions on experimental evidence where possible. Any such sensitivity of the experts' effectiveness assessments to a recent or current high profile outbreak (such as Godstone or the 2011 outbreak in Germany) may be regarded as a weakness in any expert elicitation approach. It could, however, be regarded as simply reflecting an updating of knowledge and beliefs among the panel in the light of new evidence provided by such an outbreak.

The identification of hand washing as a practical and effective means of reducing exposure concurs with the findings of the Griffin Report [Reference Griffin25]. However, the implementation of the measure raises many issues for further thought and analysis. Questions include how best to design hand-washing facilities into the farm layout and whose responsibility it is to ensure that children visiting open farms wash their hands (Griffin argues it is the parents' responsibility). Further, there is the danger that a sole measure approach could lead to the neglect of other risk factors, for example, a focus on hand washing may be at the expense of other critical control measures, for example the initial faecal contact.

BWS is best suited to the evaluation of large sets of stand-alone measures, as was the case here with 99 candidate measures. If one was taking a much smaller set of measures, with multiple levels, and trying to identify the ‘best’ bundles of those measures then alternative methods (ACA, CBC) would be more suitable. Having identified the best performing measures in the BWS study one could then consider bundling measures. Within such bundles, measures may have purely additive effects; however, in some cases they may enhance or reduce the impact of other measures. This could be assessed by modelling the cross-impact balances of intervention bundles [Reference Weimer-Jehle38].

This relates to a further notable point that, in order to keep the process focused and manageable for respondents, we focused on a single pathogen. There may, however, be spillover effects to other pathogens and policy targets. A next phase could model the potential of measures and bundles to add value through impacts on other target organisms, such as Cryptosporidium and norovirus, or contribute to the meeting of other requirements (e.g. the Water Framework Directive). This would entail re-presenting the finalized intervention list to a different panel of experts whose expertise focused on the secondary target pathogens and viruses. The same BWS methodologies would be employed but the initial research question would ask experts to evaluate the 30 interventions based upon their ability to reduce exposure to Cryptosporidium and/or norovirus.

An additional extension of the research would be to extend the sample beyond research or regulatory experts to, for example, farmers. An assessment of the consistency (or otherwise) of the assessments of the two groups would be revealing. Ultimately it will be farmers' perceptions of the practicality and effectiveness of the proposed control measures that will determine the levels of adoption in situ.

ACKNOWLEDGEMENTS

This research was undertaken as part of the project ‘Reducing E. coli O157 Risk in Rural Communities’ (RES-229-31-0003) funded under the UK Research Councils' Rural Economy and Land Use Programme. The authors thank all those who participated in the study. Advice from Dr Natasha Valeeva of Wageningen University, research assistance from Dr Seda Erdem and comments from colleagues on the project are all gratefully acknowledged.

DECLARATION OF INTEREST

None.

References

REFERENCES

1.Woolhouse, M, Gaunt, E. Ecological origins of novel human pathogens. Critical Reviews in Microbiology 2007; 33: 231242.CrossRefGoogle ScholarPubMed
2.Morgan, D, et al. Assessing the risk from emerging infections. Epidemiology and Infection 2009; 137: 15211530.CrossRefGoogle ScholarPubMed
3.Tauxe, RV. Emerging foodborne pathogens. International Journal of Food Microbiology 2002; 78: 3141.CrossRefGoogle ScholarPubMed
4.Henson, S. Estimating the incidence of food-borne Salmonella and the effectiveness of alternative control measures using the Delphi method. International Journal of Food Microbiology 1997; 35: 195204.CrossRefGoogle ScholarPubMed
5.Fish, R, et al. Unruly pathogens: eliciting values for environmental risk in the context of heterogeneous expert knowledge. Environmental Science & Policy 2009; 12: 281296.CrossRefGoogle Scholar
6.van Schaik, G, et al. Adaptive conjoint analysis to determine perceived risk factors of farmers, veterinarians and AI technicians for introduction of BHV1 to dairy farms. Preventive Veterinary Medicine 1998; 37: 101112.Google Scholar
7.Sørensen, JT, et al. Expert opinions of strategies for milk fever control. Preventive Veterinary Medicine 2002; 55: 6978.CrossRefGoogle ScholarPubMed
8.Garabed, RB, et al. Use of expert opinion for animal disease decisions: an example of foot-and-mouth disease status designation. Preventive Veterinary Medicine 2009; 92: 2030.Google Scholar
9.Valeeva, NI, et al. Improving food safety at the dairy farm level: farmers' and experts' perceptions. Review of Agricultural Economics 2005; 27: 574592.CrossRefGoogle Scholar
10.Cross, P, Williams, P, Edwards-Jones, G. Differences in the perceptions of farmers and veterinary surgeons about the efficacy of mitigation strategies for controlling bluetongue. Veterinary Record 2009; 165: 397403.CrossRefGoogle ScholarPubMed
11.Valeeva, NI, et al. Improving food safety within the dairy chain: an application of conjoint analysis. Journal of Dairy Science 2005; 88: 16011612.CrossRefGoogle ScholarPubMed
12.Orme, BK. Getting Started with Conjoint Analysis: Strategies for Product Design and Pricing Research. Madison, USA: Research Publishers, 2006.Google Scholar
13.Mennecke, BE, et al. A study of the factors that influence consumer attitudes toward beef products using the conjoint market analysis tool. Journal of Animal Science 2007; 85: 26392659.CrossRefGoogle ScholarPubMed
14.Coltman, T, Devinney, T, Keating, B. Best-worst scaling approach to predict customer choice for 3PL services. Journal of Business Logistics 2010; 32: 139152.CrossRefGoogle Scholar
15.Riley, LW, et al. Hemorrhagic colitis associated with a rare Escherichia coli serotype. New England Journal of Medicine 1983; 308: 681685.CrossRefGoogle ScholarPubMed
16.Bell, BP, et al. A multistate uutbreak of Escherichia coli O157:H7-associated bloody diarrhea and hemolytic uremic syndrome from hamburgers: the Washington experience. Journal of the American Medical Association 1994; 272: 13491353.CrossRefGoogle ScholarPubMed
17.Kassenborg, HD, et al. Farm visits and undercooked hamburgers as major risk factors for sporadic Escherichia coli O157:H7 infection: data from a case-control study in 5 FoodNet Sites. Clinical Infectious Diseases 2004; 38 (s3): S271S278.Google Scholar
18.Locking, ME, et al. Risk factors for sporadic cases of Escherichia coli O157 infection: the importance of contact with animal excreta. Epidemiology and Infection 2001; 127: 215220.CrossRefGoogle ScholarPubMed
19.O'Brien, S, Adak, G, Gilham, C. Contact with the farming environment as a major risk factor for sporadic cases of Shiga toxin-producing Escherichia coli O157 infection in humans. Emerging Infectious Diseases 2001; 7: 10491051.CrossRefGoogle Scholar
20.Parry, SM, et al. Risk factors for and prevention of sporadic infections with vero cytotoxin (shiga toxin) producing Escherichia coli O157. Lancet 1998; 351: 10191022.Google Scholar
21.Ogden, ID, et al. Long-term survival of Escherichia coli O157 on pasture following an outbreak associated with sheep at a scout camp. Letters in Applied Microbiology 2002; 34: 100104.CrossRefGoogle Scholar
22.Ogden, ID, MacRae, M, Strachan, N. Is the prevalence and shedding concentrations of E. coli O157 in beef cattle in Scotland seasonal? FEMS Microbiology Letters 2004; 233: 297300.Google Scholar
23.Khanna, R, et al. Environmental prevention of human disease from verocytotoxin-producing Escherichia coli. Nephrology Dialysis Transplantation 2008; 23: 18191822.Google Scholar
24.Solecki, O, et al. Can the high levels of human verocytotoxigenic Escherichia coli O157 infection in rural areas of NE Scotland be explained by consumption of contaminated meat? Journal of Applied Microbiology 2007; 103: 26162621.Google Scholar
25.Griffin, G. Review of the major outbreak of E. coli O157 in Surrey, 2009. Report of the Independent Investigation Committee, June 2010 (www.griffininvestigation.org.uk/).Google Scholar
26.Lynch, MF, Tauxe, RV, Hedberg, CW. The growing burden of foodborne outbreaks due to contaminated fresh produce: risks and opportunities. Epidemiology and Infection 2009; 137 (Special Issue 3): 307315.CrossRefGoogle ScholarPubMed
27.Newell, DG, et al. Food-borne diseases – the challenges of 20 years ago still persist while new ones continue to emerge. International Journal of Food Microbiology 2010; 139 (Suppl. 1): S3–S15.Google Scholar
28.Auger, P, Devinney, T, Louviere, J. Using best–worst scaling methodology to investigate consumer ethical beliefs across countries. Journal of Business Ethics 2007; 70: 299326.CrossRefGoogle Scholar
29.Marley, AAJ, Louviere, JJ. Some probabilistic models of best, worst, and best-worst choices. Journal of Mathematical Psychology 2005; 49: 464480.Google Scholar
30.Lusk, JL, Briggeman, BC. Food values. American Journal of Agricultural Economics 2009; 91: 184196.Google Scholar
31.Jaeger, SR, et al. Best-worst scaling: an introduction and initial comparison with monadic rating for preference elicitation with food products. Food Quality and Preference 2008; 19: 579588.CrossRefGoogle Scholar
32.Lee, JA, Soutar, GN, Louviere, J. Measuring values using best-worst scaling: the LOV example. Psychology and Marketing 2007; 24: 10431058.CrossRefGoogle Scholar
33.McFadden, D (ed.). Conditional Logit Analysis of Qualitative Choice Behavior. New York: Academic Press, 1974.Google Scholar
34.McFadden, D, Train, K. Mixed MNL models for discrete response. Journal of Applied Econometrics 2000; 15: 447470.Google Scholar
35.Train, K. Discrete Choice Methods and Simulation. Cambridge: Cambridge University Press, 2003.Google Scholar
36.Rigby, D, Balcombe, K, Burton, M. Distributional assumptions and mixed logit model performance: preferences and GM foods. Environmental and Resource Economics 2009; 42: 211226.Google Scholar
37.Ellis-Iversen, J, et al. Farm practices to control E. coli O157 in young cattle – a randomised controlled trial. Veterinary Research 2008; 39: 3.Google Scholar
38.Weimer-Jehle, W. Cross-impact balances: a system-theoretical approach to cross-impact analysis. Technological Forecasting and Social Change 2006; 73: 334361.Google Scholar
Figure 0

Table 1. Categories of respondent expertise invited to participate and their response rates

Figure 1

Table 2. Example of a ‘practicality’ best-worst scaling choice set

Figure 2

Table 3. List of top 30 interventions selected in round 1 for further appraisal in round 2

Figure 3

Table 4. Sample mean point estimates of the effectiveness and practicality scores for all 30 measures

Figure 4

Fig. 1. Zero-centred scatterplot of mean effectiveness and practicality scores for the 30 control measures.

Figure 5

Fig. 2. Respondent-specific practicality and effectiveness scores for two control measures (each symbol denotes the practicality and effectiveness scores of a single expert).