Hostname: page-component-8448b6f56d-c47g7 Total loading time: 0 Render date: 2024-04-17T01:42:10.385Z Has data issue: false hasContentIssue false
  • English
  • Français

On the Use of Crowdsourcing Labor Markets in Research

Published online by Cambridge University Press:  13 June 2016

Dani M. Marinova
Get access Rights & Permissions [Opens in a new window]

Abstract

Crowdsourcing platforms offer a source of inexpensive data for research. At their fingertips, researchers have a round-the-clock workforce to fill out surveys, participate in experiments, and content-analyze text, among other tasks that generate social science data and help support research. Thanks to its low cost and convenience, crowdlabor has quickly and uncritically become a mainstream tool in our discipline. While such platforms have been evaluated on their aptness to generate high-quality data, surprisingly little has been said about the economic or political implications of their usage. Among other aspects, this article problematizes the “state of legal exception” in which crowdlabor markets operate, their tendency to rely on a pool of economically vulnerable workers and the asymmetrical employment relations they create. Rather than offer an easy solution, I aim to open up a conversation about this unique set of challenges in order to acknowledge and address them.

Type
Reflections
Information
Perspectives on Politics , Volume 14 , Issue 2 , June 2016 , pp. 422 - 431
Copyright
Copyright © American Political Science Association 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Adda, Gilles, Mariani, Joseph J., Besacier, Laurent, and Gelas, Hadrien. 2013. “Economic and Ethical Background of Crowdsourcing for Speech.” In Crowdsourcing for Speech Processing, ed.Eskénazi, xine, Levow, Gina-Anne et al. . Chichester, UK: John Wiley & Sons, Ltd.Google Scholar
Ahler, Douglas J. 2014. “Self-Fulfilling Misperceptions of Public Polarization.” Journal of Politics 76(3): 607–20.Google Scholar
Arceneaux, Kevin. 2012. “Cognitive Biases and the Strength of Political Arguments.” American Journal of Political Science 56(2): 271–85.Google Scholar
Aytes, Ayhan. 2013. “Return of the Crowds: Mechacnical Turk and Neoliberal States of Exception.” In Digital Labor: The Internet as Playground and Factory, ed. Scholz, Trebor. New York: Routledge.Google Scholar
Benkler, Yochai. 2006. The Wealth of Networks: How Social Production Transforms Markets and Freedoms. New Haven, CT: Yale University Press.Google Scholar
Benoit, Kenneth, Conway, Drew, Lauderdale, Benjamin, Laver, Michael and Mikhaylov, Slava. In press. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review.Google Scholar
Berinsky, Adam J., Huber, Gregory A., and Lenz, Gabriel S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon. Com’s Mechanical Turk.” Political Analysis 20(3): 351–68.Google Scholar
Brooks, Sarah M. 2014. “Insecure Democracy: Risk and Political Participation in Brazil.” Journal of Politics 76(4): 972–85.Google Scholar
Burgoon, Brian and Dekker, Fabian. 2010. “Flexible Employment, Economic Insecurity and Social Policy Preferences in Europe.” Journal of European Social Policy 20(2): 126–41.Google Scholar
Busarovs, Aleksejs. 2013. “Ethical Aspects of Crowdsourcing, or Is It a Modern Form of Exploitation.” International Journal of Economics & Business Administration I(1): 3–14.CrossRefGoogle Scholar
Cherry, Miriam A. 2009. “Working for (Virtually) Minimum Wage: Applying the Fair Labor Standards Act in Cyberspace.” Alabama Law Review 60(5): 1077–110.Google Scholar
Clark, Andrew and Postel-Vinay, Fabien. 2009. “Job Security and Job Protection.” Oxford Economic Papers 61(2): 207–39.Google Scholar
Collins, Jane L. and Mayer, Victoria. 2010. Both Hands Tied: Welfare Reform and the Race to the Bottom in the Low-Wage Labor Market. Chicago, IL: University of Chicago Press.Google Scholar
Conway, Drew. 2013. “Methods for Collecting Large-Scale Non-Expert Text Coding.” SSRN Scholarly Paper ID 2260437 . Rochester, NY: Social Science Research Network.Google Scholar
Crawford, Jarret T. and Pilanski, Jane M.. 2014. “Political Intolerance, Right and Left.” Political Psychology 35(6): 841–51.Google Scholar
Crump, Matthew J. C., McDonnell, John V., and Gureckis, Todd M.. 2013. “Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research.” PLoS ONE 8(3): e57410.Google Scholar
Dickert, Neal and Grady, Christine. 1999. “What’s the Price of a Research Subject? Approaches to Payment for Research Participation.” New England Journal of Medicine 341(3): 198–203.Google Scholar
Doherty, David. 2013. “To Whom Do People Think Representatives Should Respond: Their District or the Country?” Public Opinion Quarterly 77(1): 237–55.Google Scholar
Doherty, David, Dowling, Conor M., and Miller, Michael G.. 2014. “Does Time Heal All Wounds? Sex Scandals, Tax Evasion, and the Passage of Time.” PS: Political Science & Politics 47(2): 357–66.Google Scholar
Dow, Steven, Kulkarni, Anand, Bunge, Brie, Nguyen, Truc, Klemmer, Scott, and Hartmann, Björn. 2011. “Shepherding the Crowd: Managing and Providing Feedback to Crowd Workers.” In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, 1669–74. CHI EA ’11. New York: ACM.Google Scholar
Fausey, Caitlin M. and Matlock, Teenie. 2011. “Can Grammar Win Elections?” Political Psychology 32(4): 563–74.Google Scholar
Felstiner, Alek. 2011. “Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry.” Berkeley Journal of Employment and Labor Law 32(1): 143–203.Google Scholar
Fort, Karën, Adda, Gilles, and Bretonnel Cohen, K.. 2011. “Amazon Mechanical Turk: Gold Mine or Coal Mine?” Computational Linguistics 37(2): 413–20.Google Scholar
Glynn, Adam N. 2013. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Public Opinion Quarterly 77(S1): 159–72.Google Scholar
Graber, Mark A. and Graber, Abraham. 2013. “Internet-based Crowdsourcing and Research Ethics: The Case for IRB Review. Journal of Medical Ethics 39(2): 115–18.Google Scholar
Grimmer, Justin, Messing, Solomon, and Westwood, Sean J.. 2012. “How Words and Money Cultivate a Personal Vote: The Effect of Legislator Credit Claiming on Constituent Credit Allocation.” American Political Science Review 106(4): 703–19.Google Scholar
Grimmer, Justin and Stewart, Brandon M.. 2013. “Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3): 267–297.Google Scholar
Guess, Andrew M. 2014. “Measure for Measure: An Experimental Test of Online Political Media Exposure.” Political Analysis 25(1): 59–75.Google Scholar
Häusermann, Silja and Schwander, Hanna. 2012. “Varieties of Dualization: Identifying Insiders and Outsiders across Regimes,” in Emmenegger, Patrick, Häusermann, Silja, Palier, Bruno, Seeleib-Kaiser, Martin (eds.). The Age of Dualization. The Changing Face of Inequality in Deindustrializing Societies. Oxford and New York: Oxford University Press.Google Scholar
Hayes, Danny, Lawless, Jennifer L., and Baitinger, Gail. 2014. “Who Cares What They Wear? Media, Gender, and the Influence of Candidate Appearance.” Social Science Quarterly 95(5): 1194–212.Google Scholar
Hardt, Michael and Negri, Antonio. 2000. Empire. Cambridge, MA: Harvard University Press.Google Scholar
Healy, Andrew and Lenz, Gabriel S.. 2014. “Substituting the End for the Whole: Why Voters Respond Primarily to the Election-Year Economy.” American Journal of Political Science 58(1): 31–47.Google Scholar
Hersh, Eitan D. and Schaffner, Brian F.. 2013. “Targeted Campaign Appeals and the Value of Ambiguity.” Journal of Politics 75(2): 520–34.Google Scholar
Hopkins, Daniel J. 2014. “The Upside of Accents: Language, Inter-Group Difference, and Attitudes toward Immigration.” British Journal of Political Science 45(3): 531–57.Google Scholar
Horton, John. 2010. “Why People Participate on Mechanical Turk, Now as a Mosaic Plot.” CrowdFlower. http://www.crowdflower.com/blog/2010/02/why-people-participate-on-mechanical-turk-now-as-a-mosaic-plot (accessed March 20, 2016).Google Scholar
Horton, John J., Rand, David G., and Zeckhauser, Richard J.. 2011. “The Online Laboratory: Conducting Experiments in a Real Labor Market.” Experimental Economics 14(3): 399–425.Google Scholar
Howe, Jeff. 2006. “The Rise of Crowdsourcing.” Wired 14(6). Retrieved from http://www.wired.com/wired/archive/14.06/crowds.html Google Scholar
Huff, Connor and Tingley, Dustin. 2014. “Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents.” Working paper, Harvard University.Google Scholar
Ipeirotis, Panagiotis G. 2010. “Demographics of Mechanical Turk.” NYU Working Paper No. CEDER-10-01. Available at SSRN, http://ssrn.com/abstract=1585030 Google Scholar
Kalleberg, Arne L., Reskin, Barbara F. and Hudson, Ken. 2000. “Bad Jobs in America: Standard and Nonstandard Employment Relations and Job Quality in the United States.” American Sociological Review 65(2): 256–78.Google Scholar
Kam, Cindy D., Wilking, Jennifer R., and Zechmeister, Elizabeth J.. 2007. “Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research.” Political Behavior 29(4): 415–40.Google Scholar
King, Desmond and Rueda, David. 2008. “Cheap Labor: The New Politics of ‘Bread and Roses’ in Industrial Democracies.” Perspectives on Politics 6(2): 279–97.Google Scholar
Kriner, Douglas L. and Shen, Francis X.. 2012. “How Citizens Respond to Combat Casualties: The Differential Impact of Local Casualties on Support for the War in Afghanistan.” Public Opinion Quarterly 76(4): 761–70.Google Scholar
Lease, Matthew, Hullman, Jessica, Bigham, Jeffrey P. et al. . 2013. “Mechanical Turk Is Not Anonymous.” SSRN paper. Available online http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2228728.Google Scholar
Leeper, Thomas. 2013. “Crowdsourcing with R and the MTurk API.” Political Methodologist 20(2): 2–7.Google Scholar
Le, John. 2010. “Amazon Mechanical Turk Survey.” Crowd Flower. http://www.crowdflower.com/blog/2010/05/amazon-mechanical-turk-survey (accessed March 20, 2016).Google Scholar
Lorey, Isabel. 2015. State of Insecurity: Government of the Precarious. London: Verso.Google Scholar
Marvit, Mshe Z. 2014. “How Crowdworkers Became the Ghosts in the Digital Machine.” The Nation, February 5.Google Scholar
Mason, Winter and Suri, Siddharth. 2012. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Behavioral Research Methods 44(1): 1–23.Google Scholar
McRobbie, Angela. 2011. “Reflections on Feminism, Immaterial Labour and the Post-Fordist Regime.” New Formations 70(1): 60–76.Google Scholar
Montgomery, Jacob M. and Cutler, Josh. 2013. “Computerized Adaptive Testing for Public Opinion Surveys.” Political Analysis 21(2): 172–92.Google Scholar
Nelson, Barbara J. 1984. “Women’s Poverty and Women’s Citizenship: Some Political Consequences of Economic Marginality.” Signs: Journal of Women in Culture and Society 10(2): 209–32.Google Scholar
Ong, Aihwa. 2006. Neoliberalism as Exception: Mutations in Citizenship and Sovereignty. Durkam, NC: Duke University Press.Google Scholar
Paolacci, Gabriele, Chandler, Jesse and Ipeirotis, Panagiotis G.. 2010. “Running Experiments on Amazon Mechanical Turk.” Judgment and Decision Making 5(5): 411–19.Google Scholar
Perlin, Ross. 2012. Intern Nation: How to Earn Nothing and Learn Little in the Brave New Economy. London: Verso.Google Scholar
Ross, Andrew. 2013. “In Search of the Lost Paycheck.” In Digital Labor: T he Internet as Playground and Factory, ed. Scholz, Trebor. New York: Routledge.Google Scholar
Ross, Joel, Irani, Lilly, Silberman, M. Six, Zaldivar, Andrew and Tomlinson, Bill. 2010. “Who Are the Crowd workers? Shifting Demographics in Mechanical Turk.” In CHI ’10 Extended Abstracts on Human Factors in Computing Systems, 2863–72. CHI EA ’10. New York: ACM.Google Scholar
Rueda, David. 2007. Social Democracy Insider Out: Partisanship & Labor Market Policy in Industrialized Democracies. Oxford: Oxford University Press.Google Scholar
Ruder, Alex I. 2014. “Institutional Design and the Attribution of Presidential Control: Insulating the President from Blame.” Quarterly Journal of Political Science 9(3): 301–35.Google Scholar
Schmidt, Florian A. 2013a. “For a Few Dollars More—Class Action against Crowdsourcing.” Peer-Reviewed Journal About# BWPWAP, (Digital Aesthetics Research Center, Aarhus University in collaboration with transmediale 2013).Google Scholar
Schmidt, Florian A. 2013b. “The Good, The Bad and the Ugly: Why Crowdsourcing Needs Ethics.” In Proceedings of the Third International Conference on Cloud and Green Computing (CGC), 531–35.Google Scholar
Schneider, Nathan. 2015. “Intellectual Piecework.” Chronicle of Higher Education, February 16.Google Scholar
Scholz, Trebor. 2013. Digital Labor: The Internet as Playground and Factory. New York: Routledge.Google Scholar
Silberman, M., Ross, Joel, Irani, Lilly, and Tomlinson, Bill. 2010. “Sellers’ Problems in Human Computation Markets.” In Proceedings of the ACM SIGKDD Workshop on Human Computation 18–21. http://dl.acm.org/citation.cfm?id=1837891 (accessed March 20, 2016).Google Scholar
Strolovitch, Dara Z. 2013. “Of Mancessions and Hecoveries: Race, Gender, and the Political Construction of Economic Crises and Recoveries.” Perspectives on Politics 11(1): 167–76.Google Scholar
Surowiecki, James. 2004. The Wisdom of Crowds: Why the Many are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. New York: Doubleday.Google Scholar
Tappscott, Don and Williams, Anthony D.. 2006. Wikinomics: How Mass Collaboration Changes Everything. New York: Portfolio.Google Scholar
Thibodeau, Paul, Peebles, Matthew M., Grodner, Daniel J., and Durgin, Frank H.. 2013. “The Wished-For Always Wins Until the Winner Was Inevitable All Along: Motivated Reasoning and Belief Bias Regulate Emotion during Elections.” Political Psychology 36(4): 431–48.Google Scholar
Tomz, Michael R. and Weeks, Jessica LP. 2013. “Public Opinion and the Democratic Peace.” American Political Science Review 107(4): 849–65.Google Scholar
U.S. Department of Health and Human Services. 2013. “Considerations and Recommendations Concerning Internet Research and Human Subjects Research Regulations, with Revisions.” Available online http://www.hhs.gov/ohrp/sachrp/commsec/attachmentbsecletter20.pdf Google Scholar
Vosko, Leah F., Martha, MacDonald, and Iain, Campbell. 2009. Gender and the Contours of Precarious Employment. New York: Routledge University Press.Google Scholar
Wolfson, Stephen M. and Lease, Matthew. 2011. “Look before You Leap: Legal Pitfalls of Crowdsourcing.” Proceedings of the American Society for Information Science and Technology 48(1): 1–10.CrossRefGoogle Scholar
Zittrain, Jonathan. 2008. “Ubiquitous Human Computing.” Philosophical Transactions Royal Society A 366(1881): 3813–21.Google Scholar