Hostname: page-component-8448b6f56d-42gr6 Total loading time: 0 Render date: 2024-04-23T06:54:46.530Z Has data issue: false hasContentIssue false
  • English
  • Français

Measurement Validity: A Shared Standard for Qualitative and Quantitative Research

Published online by Cambridge University Press:  17 January 2002

Robert Adcock  and
David Collier
Robert Adcock
Affiliation:
University of California, Berkeley
David Collier
Affiliation:
University of California, Berkeley
Get access Rights & Permissions [Opens in a new window]

Abstract

Scholars routinely make claims that presuppose the validity of the observations and measurements that operationalize their concepts. Yet, despite recent advances in political science methods, surprisingly little attention has been devoted to measurement validity. We address this gap by exploring four themes. First, we seek to establish a shared framework that allows quantitative and qualitative scholars to assess more effectively, and communicate about, issues of valid measurement. Second, we underscore the need to draw a clear distinction between measurement issues and disputes about concepts. Third, we discuss the contextual specificity of measurement claims, exploring a variety of measurement strategies that seek to combine generality and validity by devoting greater attention to context. Fourth, we address the proliferation of terms for alternative measurement validation procedures and offer an account of the three main types of validation most relevant to political scientists.

Type
Research Article
Information
American Political Science Review , Volume 95 , Issue 3 , September 2001 , pp. 529 - 546
Copyright
Copyright © American Political Science Association 2001

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alvarez, Michael, Cheibub, Jose Antonio, Limongi, Fernando, and Przeworski, Adam. 1996. “Classifying Political Regimes.” Studies in Comparative International Development 31 (Summer): 3–36.CrossRefGoogle Scholar
American Psychological Association. 1954. “Technical Recommendations for Psychological Tests and Diagnostic Techniques.” Psychological Bulletin 51 (2, Part 2): 201–38.Google Scholar
American Psychological Association. 1966. Standards for Educational and Psychological Tests and Manuals. Washington, DC: American Psychological Association.Google Scholar
Anderson, Barbara A., and Silver, Brian D.. 1986. “Measurement and Mismeasurement of the Validity of the Self-Reported Vote.” American Journal of Political Science 30 (November): 771–85.CrossRefGoogle Scholar
Anderson, Perry. 1974. Lineages of the Absolutist State. London: Verso.Google Scholar
Angoff, William H. 1988. “Validity: An Evolving Concept.” In Test Validity. ed. Howard Wainer and Henry I. Braun. Hillsdale, NJ: Lawrence Erlbaum. Pp. 19–32.Google Scholar
Babbie, Earl R. 2001. The Practice of Social Research. 9th ed. Belmont, CA: Wadsworth.Google Scholar
Bachman, Jerald G., and O’Malley, Patrick M.. 1984. “Yea-Saying, Nay-Saying, and Going to Extremes: Black-White Differences in Response Styles.” Public Opinion Quarterly 48 (Summer): 491–509.Google Scholar
Banks, Arthur S. 1979. Cross-National Time-Series Data Archive User's Manual. Binghamton: Center for Social Analysis, State University of New York at Binghamton.Google Scholar
Baumgartner, Frank R., and Walker, Jack L.. 1990. “Response to Smith's ‘Trends in Voluntary Group Membership: Comments on Baumgartner and Walker’: Measurement Validity and the Continuity of Research in Survey Research.” American Journal of Political Science 34 (August): 662–70.Google Scholar
Berry, William D., Ringquist, Evan J., Footing, Richard C., and Hanson, Russell L.. 1998. “Measuring Citizen and Government Ideology in the American States, 1960–93.” American Journal of Political Science 42 (January): 327–48.CrossRefGoogle Scholar
Bevir, Mark. 1999. The Logic of the History of Ideas. Cambridge: Cambridge University Press.Google Scholar
Bollen, Kenneth A. 1980. “Issues in the Comparative Measurement of Political Democracy.” American Sociological Review 45 (June): 370–90.CrossRefGoogle Scholar
Bollen, Kenneth A. 1989. Structural Equations with Latent Variables. New York: Wiley.CrossRefGoogle Scholar
Bollen, Kenneth A. 1990. “Political Democracy: Conceptual and Measurement Traps.” Studies in Comparative International Development 25 (Spring): 7–24.CrossRefGoogle Scholar
Bollen, Kenneth A. 1993. “Liberal Democracy: Validity and Method Factors in Cross-National Measures.” American Journal of Political Science 37 (November): 1207–30.CrossRefGoogle Scholar
Bollen, Kenneth A., Entwisle, Barbara, and Anderson, Arthur S.. 1993. “Macrocomparative Research Methods.” Annual Review of Sociology 19: 321–51.Google Scholar
Bollen, Kenneth A., and Lennox, Richard. 1991. “Conventional Wisdom on Measurement: A Structural Equation Perspective.” Psychological Bulletin 110 (September): 305–14.CrossRefGoogle Scholar
Bollen, Kenneth A., and Paxton, Pamela. 1998. “Detection and Determinants of Bias in Subjective Measures.” American Sociological Review 63 (June): 465–78.CrossRefGoogle Scholar
Bollen, Kenneth A., and Paxton, Pamela. 2000. “Subjective Measures of Liberal Democracy.” Comparative Political Studies 33 (February): 58–86.CrossRefGoogle Scholar
Brady, Henry. 1985. “The Perils of Survey Research: Inter-Personally Incomparable Responses.” Political Methodology 11 (3– 4): 269–91.Google Scholar
Brady, Henry, and Collier, David, eds. 2001. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Berkeley: Berkeley Public Policy Press, University of California, and Boulder, CO: Roman & Littlefield.Google Scholar
Brewer, John, and Hunter, Albert. 1989. Multimethod Research: A Synthesis of Styles. Newbury Park, CA: Sage.Google Scholar
Cain, Bruce E., and Ferejohn, John. 1981. “Party Identification in the United States and Great Britain.” Comparative Political Studies 14 (April): 31–47.CrossRefGoogle Scholar
Campbell, Donald T. 1960. “Recommendations for APA Test Standards Regarding Construct, Trait, or Discriminant Validity.” American Psychologist 15 (August): 546–53.CrossRefGoogle Scholar
Campbell, Donald T. 1977/1988. “Descriptive Epistemology: Psychological, Sociological, and Evolutionary.” Methodology and Epistemology for Social Science: Selected Papers. Chicago: University of Chicago Press.Google Scholar
Campbell, Donald T., and Fiske, Donald W.. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.” Psychological Bulletin 56 (March): 81–105.CrossRefGoogle ScholarPubMed
Carmines, Edward G., and Zeller, Richard A.. 1979. Reliability and Validity Assessment. Beverly Hills, CA: Sage.Google Scholar
Clausen, Aage. 1968. “Response Validity: Vote Report.” Public Opinion Quarterly 41 (Winter): 588–606.CrossRefGoogle Scholar
Collier, David. 1995. “Trajectory of a Concept: ‘Corporatism’ in the Study of Latin American Politics.” In Latin America in Comparative Perspective: Issues and Methods. ed. Smith, Peter H.. Boulder, CO: Westview. Pp. 135–62.Google Scholar
Collier, David, and Adcock, Robert. 1999. “Democracy and Dichotomies: A Pragmatic Approach to Choices about Concepts.” Annual Review of Political Science 2: 537–65.CrossRefGoogle Scholar
Collier, David, and Levitsky, Steven. 1997. “Democracy with Adjectives: Conceptual Innovation in Comparative Research.” World Politics 49 (April): 430–51.CrossRefGoogle Scholar
Collier, David, and Mahon, James E., Jr. 1993. “Conceptual ‘Stretching’ Revisited: Adapting Categories in Comparative Analysis.” American Political Science Review 87 (December): 845–55.CrossRefGoogle Scholar
Collier, Ruth Berins. 1999. Paths Toward Democracy. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Cook, Thomas D., and Campbell, Donald T.. 1979. Quasi-Experimentation. Boston: Houghton Mifflin.Google Scholar
Coppedge, Michael. 1997. “How a Large N Could Complement the Small in Democratization Research.” Paper presented at the annual meeting of the American Political Science Association, Washington, DC.Google Scholar
Coppedge, Michael, and Reinicke, Wolfgang H.. 1990. “Measuring Polyarchy.” Studies in Comparative International Development 25 (Spring): 51–72.CrossRefGoogle Scholar
Cronbach, Lee J., and Meehl, Paul E.. 1955. “Construct Validity in Psychological Tests.” Psychological Bulletin 52 (July): 281–302.CrossRefGoogle ScholarPubMed
Dahl, Robert A. 1956. A Preface to Democratic Theory. Chicago: University of Chicago Press.Google Scholar
Elkins, Zachary. 2000. “Gradations of Democracy: Empirical Tests of Alternative Conceptualizations.” American Journal of Political Science 44 (April): 293–300.CrossRefGoogle Scholar
Elster, Jon. 1999. Alchemies of the Mind. New York: Cambridge University Press.Google Scholar
Fearon, James, and Laitin, David D.. 2000. “Ordinary Language and External Validity: Specifying Concepts in the Study of Ethnicity.” Paper presented at the annual meeting of the American Political Science Association, Washington, DC.Google Scholar
Fischer, Markus. 1992. “Feudal Europe, 800–1300: Communal Discourse and Conflictual Politics.” International Organization 46 (Spring): 427–66.CrossRefGoogle Scholar
Fischer, Markus. 1993. “On Context, Facts, and Norms.” International Organization 47 (Summer): 493–500.CrossRefGoogle Scholar
Freeden, Michael. 1996. Ideologies and Political Theory: A Conceptual Approach. Oxford: Oxford University Press.Google Scholar
Gallie, W. B. 1956. “Essentially Contested Concepts.” Proceedings of the Aristotelian Society 51: 167–98.CrossRefGoogle Scholar
Gastil, Raymond D. 1988. Freedom in the World: Political Rights and Civil Liberties, 1987–1988. New York: Freedom House.Google Scholar
George, Alexander L., and Andrew Bennett. N.d. Case Studies and Theory Development. Cambridge, MA: MIT Press. Forthcoming.Google Scholar
Gerring, John. 1997. “Ideology: A Definitional Analysis.” Political Research Quarterly 50 (December): 957–94.CrossRefGoogle Scholar
Gerring, John. 1999. “What Makes a Concept Good? A Criterial Framework for Understanding Concept Formation in the Social Sciences.” Polity 31 (Spring): 357–93.Google Scholar
Gerring, John. 2001. Practical Knowledge: A Criterial Approach to Social Science Methodology. New York: Cambridge University Press.Google Scholar
Gould, Andrew C. 1999. “Conflicting Imperatives and Concept Formation.” Review of Politics 61 (Summer): 439–63.CrossRefGoogle Scholar
Green, Donald Philip. 1991. “The Effects of Measurement Error on Two-Stage Least-Squares Estimates.” In Political Analysis. vol. 2, ed. Stimson, James A.. Ann Arbor: University of Michigan Press. Pp: 57–74.Google Scholar
Green, Donald Philip, and Palmquist, Bradley L.. 1990. “Of Artifacts and Partisan Instability.” American Journal of Political Science 34 (August): 872–902.Google Scholar
Greenleaf, Eric A. 1992. “Measuring Extreme Response Style.” Public Opinion Quarterly 56 (Autumn): 382–51.Google Scholar
Guion, Robert M. 1980. “On Trinitarian Doctrines of Validity.” Professional Psychology 11 (June): 385–98.Google Scholar
Harding, Timothy, and Petras, James. 1988. “Introduction: Democratization and the Class Struggle.” Latin American Perspectives 15 (Summer): 3–17.Google Scholar
Hayduk, Leslie A. 1987. Structural Equation Modeling with LISREL. Baltimore, MD: Johns Hopkins University Press.Google Scholar
Hayduk, Leslie A. 1996. LISREL: Issues, Debates, and Strategies. Baltimore, MD: Johns Hopkins University Press.Google Scholar
Hill, Kim Q., Hanna, Stephen, and Shafqat, Sahar. 1997. “The Liberal-Conservative Ideology of U.S. Senators: A New Measure.” American Journal of Political Science 41 (October): 1395–413.CrossRefGoogle Scholar
Huntington, Samuel P. 1991. The Third Wave: Democratization in the Twentieth Century. Norman: University of Oklahoma Press.Google Scholar
Jacoby, William G. 1991. Data Theory and Dimensional Analysis. Newbury Park, CA: Sage.CrossRefGoogle Scholar
Jacoby, William G. 1999. “Levels of Measurement and Political Research: An Optimistic View.” American Journal of Political Science 43 (January): 271–301.CrossRefGoogle Scholar
Johnson, Ollie A. 1999. “Pluralist Authoritarianism in Comparative Perspective: White Supremacy, Male Supremacy, and Regime Classification.” National Political Science Review 7: 116–36.Google Scholar
Kaplan, Abraham. 1964. The Conduct of Inquiry. San Francisco: Chandler.Google Scholar
Karl, Terry Lynn. 1990. “Dilemmas of Democratization in Latin America.” Comparative Politics 22 (October): 1–21.Google Scholar
Katosh, John P., and Traugott, Michael W.. 1980. “The Consequences of Validated and Self-Reported Voting Measures.” Public Opinion Quarterly 45 (Winter): 519–35.CrossRefGoogle Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.Google Scholar
Kirk, Jerome, and Miller, Marc L.. 1986. Reliability and Validity in Qualitative Research. Beverly Hills, CA: Sage.CrossRefGoogle Scholar
Kurtz, Marcus J. 2000. “Understanding Peasant Revolution: From Concept to Theory and Case.” Theory and Society 29 (February): 93–124.CrossRefGoogle Scholar
Laitin, David D. 2000. “What Is a Language Community?” American Journal of Political Science 44 (January): 142–55.Google Scholar
Levitsky, Steven. 1998. “Institutionalization and Peronism: The Concept, the Case and the Case for Unpacking the Concept.” Party Politics 4 (1): 77–92.Google Scholar
Lijphart, Arend. 1984. Democracies: Patterns of Majoritarian and Consensus Government in Twenty-One Countries. New Haven, CT: Yale University Press.CrossRefGoogle Scholar
Lijphart, Arend. 1996. “The Puzzle of Indian Democracy: A Consociational Interpretation.” American Political Science Review 90 (June): 258–68.CrossRefGoogle Scholar
Linz, Juan J. 1975. “Totalitarian and Authoritarian Regimes.” In Handbook of Political Science. vol. 3, ed. Greenstein, Fred I. and Polsby, Nelson W.. Reading, MA: Addison-Wesley. Pp. 175–411.Google Scholar
Lipset, Seymour M. 1959. “Some Social Requisites of Democracy: Economic Development and Political Legitimacy.” American Political Science Review 53 (March): 69–105.Google Scholar
Locke, Richard, and Thelen, Kathleen. 1995. “Apples and Oranges Revisited: Contextualized Comparisons and the Study of Comparative Labor Politics.” Politics and Society 23 (September): 337–67.Google Scholar
Locke, Richard, and Thelen, Kathleen. 1998. “Problems of Equivalence in Comparative Politics: Apples and Oranges, Again.” Newsletter of the APSA Organized Section in Comparative Politics 9 (Winter): 9–12.Google Scholar
Loveman, Brian. 1994. “‘Protected Democracies’ and Military Guardianship: Political Transitions in Latin America, 1979–1993.” Journal of Interamerican Studies and World Affairs 36 (Summer): 105–89.Google Scholar
Mainwaring, Scott, Brinks, Daniel, and Aníbal Pérez-Linan. 2001. “Classifying Political Regimes in Latin America, 1945–1999.” Studies in Comparative International Development 36 (1): 37–64.CrossRefGoogle Scholar
Markoff, John. 1996. Waves of Democracy. Thousand Oaks, CA: Pine Forge.Google Scholar
Messick, Samuel. 1980. “Test Validity and the Ethics of Assessment.” American Psychologist 35 (November): 1012–27.Google Scholar
Messick, Samuel. 1989. “Validity.” In Educational Measurement. ed. Linn, Robert L.. New York: Macmillan. Pp. 13–103.Google Scholar
Moene, Karl Ove, and Wallerstein, Michael. 2000. “Inequality, Social Insurance and Redistribution.” Working Paper No. 144, Juan March Institute, Madrid, Spain.Google Scholar
Moss, Pamela A. 1992. “Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment.” Review of Educational Research 62 (Fall): 229–58.CrossRefGoogle Scholar
Moss, Pamela A. 1995. “Themes and Variations in Validity Theory.” Educational Measurement: Issues and Practice 14 (Summer): 5–13.Google Scholar
Mouffe, Chantal. 1992. Dimensions of Radical Democracy. London: Verso.Google Scholar
Nie, Norman H., Bingham Powell, G., Jr., and Prewitt, Kenneth. 1969. “Social Structure, and Political Participation: Developmental Relationships, Part I.” American Political Science Review 63 (June): 361–78.Google Scholar
Niemi, Richard, Katz, Richard, and Newman, David. 1980. “Reconstructing Past Partisanship: The Failure of the Party Identification Recall Questions.” American Journal of Political Science 24 (November): 633–51.CrossRefGoogle Scholar
O’Donnell, Guillermo. 1993. “On the State, Democratization and Some Conceptual Problems.” World Development 27 (8): 1355–69.Google Scholar
O’Donnell, Guillermo. 1996. “Illusions about Consolidation.” Journal of Democracy 7 (April): 34–51.Google Scholar
Orum, Anthony M., Feagin, Joe R., and Sjoberg, Gideon. 1991. “Introduction: The Nature of the Case Study.” In A Case for the Case Study. ed. Feagin, Joe R., Orum, Anthony M., and Sjoberg, Gideon. Chapel Hill: University of North Carolina Press. Pp. 1–26.Google Scholar
Paxton, Pamela. 2000. “Women in the Measurement of Democracy: Problems of Operationalization.” Studies in Comparative International Development 35 (Fall): 92–111.CrossRefGoogle Scholar
Pitkin, Hanna Fenichel. 1967. The Concept of Representation. Berkeley: University of California Press.Google Scholar
Pitkin, Hanna Fenichel. 1987. “Rethinking Reification.” Theory and Society 16 (March): 263–93.Google Scholar
Przeworski, Adam, Alvarez, Michael, Cheibub, Jose Antonio, and Limongi, Fernando. 1996. “What Makes Democracies Endure?” Journal of Democracy 7 (January): 39–55.Google Scholar
Przeworski, Adam, and Teune, Henry. 1970. Logic of Comparative Social Inquiry. New York: John Wiley.Google Scholar
Rabkin, Rhoda. 1992. “The Aylwin Government and ‘Tutelary’ Democracy: A Concept in Search of a Case?” Journal of Interamerican Studies and World Affairs 34 (Winter): 119–94.Google Scholar
Ragin, Charles C. 1987. The Comparative Method. Berkeley: University of California Press.Google Scholar
Ragin, Charles C. 1994. Constructing Social Research. Thousand Oaks, CA: Pine Forge.Google Scholar
Reus-Smit, Christian. 1997. “The Constitutional Structure of International Society and the Nature of Fundamental Institutions.” International Organization 51 (Autumn): 555–89.Google Scholar
Ruggie, John G. 1998. “What Makes the World Hang Together? Neo-Utilitarianism and the Social Constructivist Challenge.” International Organization 52 (Autumn): 855–85.Google Scholar
Russett, Bruce. 1993. Grasping the Democratic Peace. Princeton, NJ: Princeton University Press.Google Scholar
Sartori, Giovanni. 1970. “Concept Misformation in Comparative Research.” American Political Science Review 64 (December): 1033–53.Google Scholar
Sartori, Giovanni. 1975. “The Tower of Babel.” In Tower of Babel, ed. Giovanni Sartori, Fred W. Riggs, and Henry Teune. International Studies Association, University of Pittsburgh. Pp. 7–37.Google Scholar
Sartori, Giovanni, ed. 1984. Social Science Concepts: A Systematic Analysis. Beverly Hills, CA: Sage.Google Scholar
Sartori, Giovanni, Riggs, Fred W., and Teune, Henry. 1975. Tower of Babel: On the Definition and Analysis of Concepts in the Social Sciences. International Studies Association, University of Pittsburgh.Google Scholar
Schaffer, Frederic Charles. 1998. Democracy in Translation: Understanding Politics in an Unfamiliar Culture. Ithaca, NY: Cornell University Press.Google Scholar
Schmitter, Philippe C., and Karl, Terry Lynn. 1992. “The Types of Democracy Emerging in Southern and Eastern Europe and South and Central America.” In Bound to Change: Consolidating Democracy in East Central Europe. ed. Volten, Peter M. E.. New York: Institute for EastWest Studies. Pp. 42–68.Google Scholar
Schrodt, Philip A., and Gerner, Deborah J.. 1994. “Validity Assessment of a Machine-Coded Event Data Set for the Middle East, 1982–92.” American Journal of Political Science 38 (August): 825–54.Google Scholar
Schumpeter, Joseph. 1947. Capitalism, Socialism and Democracy. New York: Harper.Google Scholar
Shepard, Lorrie. 1993. “Evaluating Test Validity.” Review of Research in Education 19: 405–50.Google Scholar
Shively, W. Phillips. 1998. The Craft of Political Research. Upper Saddle River, NJ: Prentice Hall.Google Scholar
Shultz, Kenneth S., Riggs, Matt L., and Kottke, Janet L.. 1998. “The Need for an Evolving Concept of Validity in Industrial and Personnel Psychology: Psychometric, Legal, and Emerging Issues.” Current Psychology 17 (Winter): 265–86.Google Scholar
Sicinski, Andrzej. 1970. “‘Don't Know’ Answers in Cross-National Surveys.” Public Opinion Quarterly 34 (Spring): 126–9.Google Scholar
Sireci, Stephen G. 1998. “The Construct of Content Validity.” Social Indicators Research 45 (November): 83–117.CrossRefGoogle Scholar
Skocpol, Theda. 1992. Protecting Soldiers and Mothers. Cambridge, MA: Harvard University Press.Google Scholar
Sussman, Leonard R. 1982. “The Continuing Struggle for Freedom of Information.” In Freedom in the World. ed. Gastil, Raymond D.. Westport, CT: Greenwood. Pp. 101–19.Google Scholar
Valenzuela, J. Samuel. 1992. “Democratic Consolidation in Post-Transitional Settings: Notion, Process, and Facilitating Conditions.” In Issues in Democratic Consolidation. ed. Scott Mainwaring, Guillermo O’Donnell, and J. Samuel Valenzuela. Notre Dame, IN: University of Notre Dame Press. Pp. 57–104.Google Scholar
Vanhanen, Tatu. 1979. Power and the Means to Power. Ann Arbor, MI: University Microfilms International.Google Scholar
Vanhanen, Tatu. 1990. The Process of Democratization. New York: Crane Russak.Google Scholar
Verba, Sidney. 1967. “Some Dilemmas in Comparative Research.” World Politics 20 (October): 111–27.Google Scholar
Verba, Sidney. 1971. “Cross-National Survey Research: The Problem of Credibility.” In Comparative Methods in Sociology. ed. Ivan Vallier. Berkeley: University of California Press. Pp. 309–56.Google Scholar
Verba, Sidney, Kelman, Steven, Orren, Gary R., Miyake, Ichiro, Watanuki, Joji, Kabashima, Ikuo, and G. Donald Ferree, Jr. 1987. Elites and the Idea of Equality. Cambridge, MA: Harvard University Press.Google Scholar
Verba, Sidney, Nie, Norman, and Kim, Jae-On. 1978. Participation and Political Equality. Cambridge: Cambridge University Press.Google Scholar
Webb, Eugene J., Campbell, Donald T., Schwartz, Richard D., and Sechrest, Lee. 1966. Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally.Google Scholar
Zeitsch, John, Lawrence, Denis, and Salernian, John. 1994. “Comparing Like with Like in Productivity Studies: Apples, Oranges and Electricity.” Economic Record 70 (June): 162–70.Google Scholar
Zeller, Richard A., and Carmines, Edward G.. 1980. Measurement in the Social Sciences: The Link between Theory and Data. Cambridge: Cambridge University Press.Google Scholar