Hostname: page-component-8448b6f56d-cfpbc Total loading time: 0 Render date: 2024-04-23T08:50:28.189Z Has data issue: false hasContentIssue false
  • English
  • Français

Reliability and Consensus of Experienced Wine Judges: Expertise Within and Between?

Published online by Cambridge University Press:  31 July 2012

Robert H. Ashton
Robert H. Ashton
Affiliation:
L. Palmer Fox Professor, Fuqua School of Business, Duke University.
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper considers the levels of reliability and consensus of wine quality judgments found in studies of experienced wine judges. Both reliability, which concerns the similarity of repeat judgments of a particular wine by the same judge, and consensus, which concerns the similarity of judgments of a particular wine across judges, are necessary requirements for expertise in wine judging. Reliability and consensus levels found in wine judging are compared to those documented by a large body of research in six other fields: medicine, clinical psychology, business, auditing, personnel management, and meteorology. In all fields, including wine judging, reliability is greater than consensus. Both reliability and consensus are, on average, substantially lower in wine judging than in other fields, although tremendous variability exists across judges in every field. Overall, little support is found for the idea that experienced wine judges should be regarded as experts. (JEL Classification: C91)

Type
Research Article
Information
Journal of Wine Economics , Volume 7 , Issue 1 , May 2012 , pp. 70 - 87
Copyright
Copyright © American Association of Wine Economists 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Amerine, M.A., and Roessler, E.B. (1983). Wines: Their sensory evaluation. New York: W.H. Freeman.Google Scholar
Arens, A.A., Elder, R.J., and Beasley, M.S. (2005). Auditing and assurance services. Upper Saddle River, NJ: Prentice Hall.Google Scholar
Ashenfelter, O., and Quandt, R. (1999). Analyzing a wine tasting statistically. Chance, 12, 16–20.CrossRefGoogle Scholar
Ashton, A.H. (1985). Does consensus imply accuracy in accounting studies of decision making? Accounting Review, 60, 173–185.Google Scholar
Ashton, R.H. (1999). Enriching the “expertise paradigm” of accounting research: Conscientiousness, general cognitive ability, and goal orientation. Advances in Accounting Behavioral Research, 2, 3–14.Google Scholar
Ashton, R.H. (2000). A review and analysis of research on the test-retest reliability of professional judgment. Journal of Behavioral Decision Making, 13, 277–294.3.0.CO;2-B>CrossRefGoogle Scholar
Ashton, R.H. (2011). Improving experts’ wine quality judgments: Two heads are better than one. Journal of Wine Economics, 6, 160–178.CrossRefGoogle Scholar
Ashton, R.H., and Ashton, A.H. (1995). Perspectives on judgment and decision-making research in accounting and auditing. In Ashton, R.H. and Ashton, A.H. (Eds.), Judgment and decision-making research in accounting and auditing. New York: Cambridge University Press. Pages 3–5.CrossRefGoogle Scholar
Baker, G.A., and Amerine, M.A. (1953). Organoleptic ratings of wines estimated from analytical data. Food Research, 18, 381–389.CrossRefGoogle Scholar
Bartoshuk, L.M. (1993). The biological basis of food perception and acceptance. Food Quality and Preference, 4, 21–32.CrossRefGoogle Scholar
Bédard, J., and Chi, M.T.H. (1993). Expertise in auditing. Auditing: A Journal of Practice & Theory, 12(Supplement), 21–45.Google Scholar
Bouwman, M.J., and Bradley, W.E. (1997). Judgment and decision making, part II: Expertise, consensus and accuracy. In Arnold, V. and Sutton, S.G. (Eds.), Behavioral accounting research: Foundations and frontiers. Sarasota, FL: American Accounting Association. Pages 89–133.Google Scholar
Brien, C.J., May, P., and Mayo, O. (1987). Analysis of judge performance in wine-quality evaluations. Journal of Food Science, 52, 1273–1279.CrossRefGoogle Scholar
Broomell, S.B., and Budescu, D.V. (2009). Why are experts correlated? Decomposing correlations between judges. Psychometrika, 74, 531–553.CrossRefGoogle Scholar
Cicchetti, D.V. (2004a). Who won the 1976 blind tasting of French Bordeaux and U.S. cabernets? Parametrics to the rescue. Journal of Wine Research, 15, 211–220.CrossRefGoogle Scholar
Cicchetti, D.V. (2004b). On designing experiments and analysing data to assess the reliability and accuracy of blind wine tastings. Journal of Wine Research, 15, 221–226.CrossRefGoogle Scholar
Cicchetti, D.V. (2006a). The Paris 1976 wine tastings revisited once more: Comparing ratings of consistent and inconsistent tasters. Journal of Wine Economics, 1, 125–140.CrossRefGoogle Scholar
Cicchetti, D.V. (2006b). The 1976 blind wine tastings: On the consistency of tasters from chardonnays to cabernets. Vineyard Data Quantification Society (www.vdqs.net).Google Scholar
Cliff, M.A., and King, M.C. (1997). The evaluation of judges at wine competitions: The application of eggshell plots. Journal of Wine Research, 8, 75–80.CrossRefGoogle Scholar
Cooksey, R.W. (1996). Judgment analysis. San Diego: Academic Press.Google Scholar
Davis, E.B., Kennedy, S.J., and Maines, L.A. (2000). The relation between consensus and accuracy in low-to-moderate accuracy tasks: An auditing example. Auditing: A Journal of Practice & Theory, 19, 101–121.CrossRefGoogle Scholar
Detre, K.M., Wright, E., Murphy, M.L., and Takaro, T. (1975). Observer agreement in evaluating coronary angiograms. Circulation, 52, 979–986.CrossRefGoogle ScholarPubMed
Einhorn, H.J. (1974). Expert judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59, 562–571.CrossRefGoogle Scholar
Einhorn, H.J., and Hogarth, R.M. (1981). Rationality and the sanctity of competence. Behavioral and Brain Sciences, 4, 334–335.CrossRefGoogle Scholar
Fischhoff, B. (1982). Debiasing. In Kahneman, D., Slovic, P., and Tversky, A. (Eds.), Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press. Pages 422–444.CrossRefGoogle Scholar
Gawel, R., and Godden, P.W. (2008). Evaluation of the consistency of wine quality assessments from expert wine tasters. Australian Journal of Grape and Wine Research, 14, 1–8.CrossRefGoogle Scholar
Gawel, R., Royal, A., and Leske, P. (2002). The effect of different oak types on the sensory properties of Chardonnay. Australian and New Zealand Wine Industry Journal, 17, 14–20.Google Scholar
Ghiselli, E.E. (1964). Theory of psychological measurement. New York: McGraw-Hill.Google Scholar
Goldberg, L.R. (1970). Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. Psychological Bulletin, 73, 422–432.Google Scholar
Goldwyn, C., and Lawless, H. (1991). How to taste wine (for fun and profit). ASTM Standardization News, 19, 32–37.Google Scholar
Goode, J. (2008). Experiencing wine: Why critics mess up (some of the time). In Allhoff, F. (Ed.), Wine & philosophy: A symposium on thinking and drinking. Malden, MA: Blackwell. Pages 137–153.Google Scholar
Hodgson, R.T. (2008). An examination of judge reliability at a major U.S. wine competition. Journal of Wine Economics, 3, 105–113.CrossRefGoogle Scholar
Hodgson, R.T. (2009a). An analysis of the concordance among 13 U.S. wine competitions. Journal of Wine Economics, 4, 1–9.CrossRefGoogle Scholar
Hodgson, R.T. (2009b). How expert are “expert” wine judges? Journal of Wine Economics, 4, 233–241.CrossRefGoogle Scholar
Hulkower, N. (2009). The judgment of Paris according to Borda. Journal of Wine Research, 20, 171–182.CrossRefGoogle Scholar
Karelaia, N., and Hogarth, R.M. (2008). Determinants of linear judgment: A meta-analysis of lens model studies. Psychological Bulletin, 134, 404–426.CrossRefGoogle ScholarPubMed
Kaufmann, E., and Athanasou, J.A. (2009). A meta-analysis of judgment achievement as defined by the lens model equation. Swiss Journal of Psychology, 68, 99–112.CrossRefGoogle Scholar
Keasey, K., and Watson, R. (1989). Consensus and accuracy in accounting studies of decision making: A note on a new measure of consensus. Accounting, Organizations and Society, 14, 337–345.CrossRefGoogle Scholar
Kenny, D.A. (1991). A general model of consensus and accuracy in interpersonal perception. Psychological Review, 98, 155–163.CrossRefGoogle ScholarPubMed
Lawless, H., Liu, Y., and Goldwyn, C. (1997). Evaluation of wine quality using a small-panel hedonic scaling method. Journal of Sensory Studies, 12, 317–332.CrossRefGoogle Scholar
Lee, J.W., and Yates, J.F. (1992). How quantity judgment changes as the number of cues increases: An analytical framework and review. Psychological Bulletin, 12, 363–377.CrossRefGoogle Scholar
Lindley, D.V. (2006). Analysis of a wine tasting. Journal of Wine Economics, 1, 33–41.CrossRefGoogle Scholar
Lord, F., and Novick, M. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
Pincus, K.V. (1990). Audit judgment consensus: A model for dichotomous decisions. Auditing: A Journal of Practice & Theory, 9, 1–20.Google Scholar
Quandt, R.E. (2006). Measurement and inference in wine tasting. Journal of Wine Economics, 1, 7–30.CrossRefGoogle Scholar
Quandt, R.E. (2007). A note on a test for the sum of ranksums. Journal of Wine Economics, 2, 98–102.CrossRefGoogle Scholar
Schmidt, F.L., and Hunter, J.E. (1992). Development of a causal model of processes determining job performance. Current Directions in Psychological Science, 1, 89–92.Google Scholar
Shanteau, J. (2001). What does it mean when experts disagree? In Salas, E. and Klein, G. (Eds.), Linking expertise and naturalistic decision making. Hillsdale, NJ: Erlbaum. Pages 229–244.Google Scholar
Taber, G.M. (2006). Judgment of Paris: California vs. France and the historic 1976 Paris tasting that revolutionized wine. New York: Scribner.Google Scholar
Weiss, D.J., and Shanteau, J. (2004). The vice of consensus and the virtue of consistency. In Smith, K., Shanteau, J., and Johnson, P. (Eds.), Psychological investigations of competence in decision making. Cambridge: Cambridge University Press. Pages 226–240.Google Scholar
Wright, W.F. (1988). Audit judgment consensus and experience. In Ferris, K.R. (Ed.), Behavioral accounting research: A critical analysis. Columbus, OH: Century VII. Pages 305–328.Google Scholar