Hostname: page-component-8448b6f56d-tj2md Total loading time: 0 Render date: 2024-04-17T19:22:25.270Z Has data issue: false hasContentIssue false
  • English
  • Français

Wisdom of crowds versus wisdom of linguists – measuring the semantic relatedness of words

Published online by Cambridge University Press:  09 September 2009

TORSTEN ZESCH  and
IRYNA GUREVYCH
TORSTEN ZESCH
Affiliation:
Ubiquitous Knowledge Processing Lab, Computer Science Department, Technische Universität Darmstadt, Hochschulstr. 10, 64289 Darmstadt, Germany e-mail: zesch@tk.informatik.tu-darmstadt.de, gurevych@tk.informatik.tu-darmstadt.de
IRYNA GUREVYCH
Affiliation:
Ubiquitous Knowledge Processing Lab, Computer Science Department, Technische Universität Darmstadt, Hochschulstr. 10, 64289 Darmstadt, Germany e-mail: zesch@tk.informatik.tu-darmstadt.de, gurevych@tk.informatik.tu-darmstadt.de
Get access Rights & Permissions [Opens in a new window]

Abstract

In this article, we present a comprehensive study aimed at computing semantic relatedness of word pairs. We analyze the performance of a large number of semantic relatedness measures proposed in the literature with respect to different experimental conditions, such as (i) the datasets employed, (ii) the language (English or German), (iii) the underlying knowledge source, and (iv) the evaluation task (computing scores of semantic relatedness, ranking word pairs, solving word choice problems). To our knowledge, this study is the first to systematically analyze semantic relatedness on a large number of datasets with different properties, while emphasizing the role of the knowledge source compiled either by the ‘wisdom of linguists’ (i.e., classical wordnets) or by the ‘wisdom of crowds’ (i.e., collaboratively constructed knowledge sources like Wikipedia).

The article discusses benefits and drawbacks of different approaches to evaluating semantic relatedness. We show that results should be interpreted carefully to evaluate particular aspects of semantic relatedness. For the first time, we employ a vector based measure of semantic relatedness, relying on a concept space built from documents, to the first paragraph of Wikipedia articles, to English WordNet glosses, and to GermaNet based pseudo glosses. Contrary to previous research (Strube and Ponzetto 2006; Gabrilovich and Markovitch 2007; Zesch et al. 2007), we find that ‘wisdom of crowds’ based resources are not superior to ‘wisdom of linguists’ based resources. We also find that using the first paragraph of a Wikipedia article as opposed to the whole article leads to better precision, but decreases recall. Finally, we present two systems that were developed to aid the experiments presented herein and are freely available1 for research purposes: (i) DEXTRACT, a software to semi-automatically construct corpus-driven semantic relatedness datasets, and (ii) JWPL, a Java-based high-performance Wikipedia Application Programming Interface (API) for building natural language processing (NLP) applications.

Type
Papers
Information
Natural Language Engineering , Volume 16 , Issue 1 , January 2010 , pp. 25 - 59
Copyright
Copyright © Cambridge University Press 2009

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anscombe, F. J. 1973. Graphs in statistical analysis. American Statistician 27: 17–21.CrossRefGoogle Scholar
Banerjee, S., and Pedersen, T. 2002. An adapted lesk algorithm for word sense disambiguation using WordNet. In CICLing '02: Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing, pp. 136–145, London: Springer Verlag.CrossRefGoogle Scholar
Bernard, J. 1986. The Macquarie Thesaurus. Sidney, Australia: Macquarie Library.Google Scholar
Boyd-Graber, J., Fellbaum, C., Osherson, D., and Shapire, R. 2006. Adding dense, weighted, connections to WordNet. In Proceedings of the Third Global WordNet Meeting, Jeju Island, Korea.Google Scholar
Budanitsky, A., and Hirst, G. 2006. Evaluating WordNet-based measures of semantic distance. Computational Linguistics 32 (1): 13–47.CrossRefGoogle Scholar
Fellbaum, C. 1998. WordNet an Electronic Lexical Database. Cambridge, MA: MIT Press.CrossRefGoogle Scholar
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., and Wolfman, G. 2002. Placing search in context: the concept revisited. ACM Transactions on Information Systems 20 (1): 116–31.Google Scholar
Gabrilovich, E., and Markovitch, S. 2007. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), pp. 1606–11, Hyderabad, India.Google Scholar
Galley, M., and McKeown, K. 2003. Improving word sense disambiguation in lexical chaining. In Proceedings of 18th International Joint Conference on Artificial Intelligence (IJCAI'03), pp. 1486–8, Acapulco, Mexico.Google Scholar
Gurevych, I. 2005. Using the structure of a conceptual network in computing semantic relatedness. In Proceedings of the 2nd International Joint Conference on Natural Language Processing, pp. 767–78. Jeju Island, Korea.Google Scholar
Gurevych, I., Müller, C., and Zesch, T. 2007. What to be? – electronic career guidance based on semantic relatedness. In Proceedings of ACL, pp. 1032–9, Prague, Czech Republic. Association for Computational Linguistics.Google Scholar
Gurevych, I., and Strube, M. 2004. Semantic similarity applied to spoken dialogue summarization. In The 22nd International Conference on Computational Linguistics (COLING), pp. 764–70, Geneva, Switzerland.CrossRefGoogle Scholar
Halliday, M. A. K., and Hasan, R. 1976. Cohesion in English. London: Longman.Google Scholar
Hirst, G., and St-Onge, D. 1998. Lexical chains as representation of context for the detection and correction malapropisms. In Fellbaum, Christiane (ed.), WordNet: An Electronic Lexical Database and Some of Its Applications, pp. 305–332. Cambridge, MA: The MIT Press.Google Scholar
Jarmasz, M., and Szpakowicz, S. 2003. Roget's thesaurus and semantic similarity. In Proceedings of Recent Advances in Natural Language Processing, pp. 111–20.Google Scholar
Jiang, J. J., and Conrath, D. W. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th International Conference on Research in Computational Linguistics, Taipei, Taiwan.Google Scholar
Kozima, H., and Furugori, T. 1993. Similarity between words computed by spreading activation on an English dictionary. In Proceedings of the sixth conference of the European chapter of the Association for Computational Linguistics, pp. 232–9, Morristown, NJ.Google Scholar
Kunze, C. 2004. Computerlinguistik und Sprachtechnologie. In Carstensen, K. U., Ebert, C., Endriss, C., Jekat, S., Klabunde, R., and Langer, H. (eds.), Lexikalisch-semantische Wortnetze, pp. 423–31. Berlin: Spektrum Akademischer Verlag.Google Scholar
Leacock, C., and Chodorow, M. 1998. Combining local context and WordNet similarity for word sense identification. In WordNet: An Electronic Lexical Database, pp. 265–83. Cambridge, MA: MIT Press.Google Scholar
Lesk, M. 1986. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. In Proceedings of the 5th Annual International Conference on Systems Documentation, pp. 24–6. Toronto, Canada.Google Scholar
Li, Y., Bandar, Z. A., and McLean, D. 2003. An approach for measuring semantic similarity between words using multiple information sources. IEEE Transactions on Knowledge and Data Engineering 15: 871–82.Google Scholar
Lin, D. 1998. An information-theoretic definition of similarity. In Proceedings of International Conference on Machine Learning, pp. 296–304. Madison, WI.Google Scholar
McHale, M. 1998. A comparison of wordnet and roget's taxonomy for measuring semantic similarity. CoRR, cmp-lg/9809003.Google Scholar
Mihalcea, R., and Moldovan, D. I. 1999. A method for word sense disambiguation of unrestricted text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pp. 152–8, Maryland, MD: Association for Computational Linguistics.Google Scholar
Miller, G. A., and Charles, W. G. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes 6 (1): 1–28.CrossRefGoogle Scholar
Mohammad, S., Gurevych, I., Hirst, G., and Zesch, T. 2007. Cross-lingual distributional profiles of concepts for measuring semantic distance. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 571–80, Prague, Czech Republic: Association for Computational Linguistics.Google Scholar
Morris, J., and Hirst, G. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics 17 (1): 21–48.Google Scholar
Morris, J., and Hirst, G. 2004. Non-classical lexical semantic relations. In Workshop on Computational Lexical Semantics, Human Language Technology Conference of the North American Chapter of the ACL, pp. 46–51. Boston, MA.Google Scholar
Patwardhan, S., Banerjee, S., and Pedersen, T. 2003. Using measures of semantic relatedness for word sense disambiguation. In Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, pp. 241–57, Mexico City, Mexico.CrossRefGoogle Scholar
Patwardhan, S., and Pedersen, T. 2006. Using WordNet based context vectors to estimate the semantic relatedness of concepts. In Proceedings of the EACL 2006 Workshop Making Sense of Sense - Bringing Computational Linguistics and Psycholinguistics Together, pp. 1–8, Trento, Italy: Association for Computational Linguistics.Google Scholar
Pirro, G., and Seco, N. 2008). Design, implementation and evaluation of a new semantic similarity metric combining features and intrinsic information content. In OTM '08: Proceedings of the OTM 2008 Confederated International Conferences, CoopIS, DOA, GADA, IS, and ODBASE, pp. 1271–88, Monterrey, Mexico.Google Scholar
Procter, P. 1978. Longman Dictionary of Contemporary English. Longman, London.Google Scholar
Qiu, Y., and Frei, H. P. 1993. Concept based query expansion. In Proceedings of the 16th ACM International Conference on Research and Development in Information Retrieval, ACMGoogle Scholar
Rada, R., Mili, H., Bicknell, E., and Blettner, M. 1989. Development and application of a metric on semantic nets. IEEE Transactions on Systems, Man, and Cybernetics 19 (1): 17–30.CrossRefGoogle Scholar
Resnik, P. 1995 Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pp. 448–53, Montreal, Canada.Google Scholar
Roget, P. 1962. Roget's International Thesaurus, 3rd ed. Berrey, L. V., and Carruth, G. (eds.), New York: Thomas Y. Crowell Co.Google Scholar
Rubenstein, H., and Goodenough, J. B. 1965. Contextual correlates of synonymy. Communications of the ACM 8 (10): 627–33.CrossRefGoogle Scholar
Salton, G., and McGill, M. J. 1983. Introduction to Modern Information Retrieval. New York: McGraw-Hill.Google Scholar
Seco, N., and Hayes, T. V. J. 2004. An intrinsic information content metric for semantic similarity in WordNet. In Proceedings of ECAI'2004, the 16th European Conference on Artificial Intelligence, Valencia, Spain.Google Scholar
Silber, H. G., and McCoy, K. F. 2002. Efficiently computed lexical chains as an intermediate representation for automatic text summarization. Comput. Linguist. 28 (4): 487–96.CrossRefGoogle Scholar
Stevenson, M., and Greenwood, M. A. 2005. A semantic approach to ie pattern induction. In ACL '05: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 379–86. Morristown, NJ: Association for Computational Linguistics.CrossRefGoogle Scholar
Strube, M., and Ponzetto, S. P. 2006. WikiRelate! Computing semantic relatedness using Wikipedia. In Proceedings of the 21st National Conference on ArtificialIntelligence (AAAI-06), pp. 1419–24, Boston, MA.Google Scholar
Turney, P. 2006. Expressing implicit semantic relations without supervision. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL, pp. 313–20, Sydney, Australia: Association for Computational Linguistics.Google Scholar
Voss, J. 2006. Collaborative thesaurus tagging the Wikipedia way. CoRR, abs/cs/0604036.Google Scholar
Wallace, D., and Wallace, L. A. 2001–2005. Reader's Digest, das Beste für Deutschland. January 2001–December 2005. Stuttgart: Verlag Das Beste.Google Scholar
Weeds, J. E. 2003. Measures and Applications of Lexical Distributional Similarity. PhD thesis, East Sussex, UK: University of Sussex.Google Scholar
Wu, Z., and Palmer, M. 1994. Verb semantics and lexical selection. In 32nd Annual Meeting of the ACL, pp. 133–8, Las Cruces, Mexico: Association for Computational Linguistics.Google Scholar
Yang, D., and Powers, D. M. W. 2006. Verb similarity on the taxonomy of WordNet. In Proceedings of the Third International WordNet Conference (GWC-06), pp. 121–8, Jeju Island, Korea.Google Scholar
Zesch, T., and Gurevych, I. 2006. Automatically creating datasets for measures of semantic relatedness. In Proceedings of the ACL-Workshop on Linguistic Distances, pp. 16–24, Sydney, Australia: Association for Computational Linguistics.CrossRefGoogle Scholar
Zesch, T., and Gurevych, I. 2007. Analysis of the Wikipedia category graph for NLP applications. In Proceedings of the TextGraphs-2 Workshop (NAACL-HLT 2007), pp. 1–8, Rochester, NY. Association for Computational Linguistics.Google Scholar
Zesch, T., Gurevych, I., and Mühlhäuser, M. 2007a. Analyzing and accessing Wikipedia as a lexical semantic resource. In Rehm, G., Witt, A., and Lemnitzer, L. (eds.), Data Structures for Linguistic Resources and Applications, pp. 197–205. Tuebingen, Germany: Gunter Narr.Google Scholar
Zesch, T., Gurevych, I., and Mühlhäuser, M. 2007b. Comparing Wikipedia and german wordnet by evaluating semantic relatedness on multiple datasets. In Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 205–8. Rochester, NY: Association for Computational Linguistics.Google Scholar
Zesch, T., Müller, C., and Gurevych, I. 2008a. Extracting lexical semantic knowledge from Wikipedia and Wiktionary. In Proceedings of the Conference on Language Resources and Evaluation (LREC), Marrakech, Morocco.Google Scholar
Zesch, T., Müller, C., and Gurevych, I. 2008b Using Wiktionary for computing semantic relatedness. In Proceedings of AAAI, pp. 861–7. Chicago, IL.Google Scholar