A dependency-based method for evaluating broad-coverage parsers
DEKANG LIN a1 a1 Department of Computer Science, University of Manitoba, Winnipeg, Manitoba, Canada R3T 2N2; e-mail: lindek@cs.umanitoba.ca
Abstract
With the emergence of broad-coverage parsers, quantitative evaluation
of parsers becomes
increasingly more important. We propose a dependency-based method for evaluating
broad-coverage parsers that offers more meaningful performance
measures than previous approaches.
We also present a structural pattern-matching mechanism that can be used
to eliminate
inconsequential differences among different parse trees. Previous evaluation
methods have
only evaluated the overall performance of parsers. The dependency-based
method can also
evaluate parsers with respect to different kinds of grammatical relationships
or different types
of lexical categories. An algorithm for transforming constituency trees
into dependency trees
is presented, which makes the evaluation method applicable to both constituency
grammars
and dependency grammars.