Hostname: page-component-8448b6f56d-gtxcr Total loading time: 0 Render date: 2024-04-23T16:41:28.275Z Has data issue: false hasContentIssue false
  • English
  • Français

The Bilingual Language Interaction Network for Comprehension of Speech*

Published online by Cambridge University Press:  06 September 2012

ANTHONY SHOOK  and
VIORICA MARIAN
ANTHONY SHOOK*
Affiliation:
Northwestern University
VIORICA MARIAN
Affiliation:
Northwestern University
*
Address for correspondence: Anthony Shook, Department of Communication Sciences and Disorders, Northwestern University, 2240 Campus Drive, Evanston, IL 60208, USAa-shook@northwestern.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

During speech comprehension, bilinguals co-activate both of their languages, resulting in cross-linguistic interaction at various levels of processing. This interaction has important consequences for both the structure of the language system and the mechanisms by which the system processes spoken language. Using computational modeling, we can examine how cross-linguistic interaction affects language processing in a controlled, simulated environment. Here we present a connectionist model of bilingual language processing, the Bilingual Language Interaction Network for Comprehension of Speech (BLINCS), wherein interconnected levels of processing are created using dynamic, self-organizing maps. BLINCS can account for a variety of psycholinguistic phenomena, including cross-linguistic interaction at and across multiple levels of processing, cognate facilitation effects, and audio-visual integration during speech comprehension. The model also provides a way to separate two languages without requiring a global language-identification system. We conclude that BLINCS serves as a promising new model of bilingual spoken language comprehension.

Keywords

spoken language comprehensionmodeling speech processingconnectionist modelsself-organizing mapslanguage interaction
Type
Research Article
Information
Bilingualism: Language and Cognition , Volume 16 , Issue 2: Computational Modeling of Bilingualism , April 2013 , pp. 304 - 324
Copyright
Copyright © Cambridge University Press 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

The authors would like to thank the members of the Northwestern Bilingualism and Psycholinguistics Laboratory, as well as Dr. Ping Li and two anonymous reviewers for their helpful comments. This research was funded in part by grant R01HD059858 to the second author, and the John D. and Lucille H. Clarke Scholarship to the first author.

References

Allopenna, P., Magnuson, J., & Tanenhaus, M. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38, 419–439.Google Scholar
Ameel, E., Storms, G., Malt, B., & Sloman, S. (2005). How bilinguals solve the naming problem. Journal of Memory and Language, 52, 309–329.Google Scholar
Bartolotti, J., & Marian, V. (2012). Language learning and control in monolinguals and bilinguals. Cognitive Science, 36 (6), 1129–1147.Google Scholar
Bates, E., Devescovi, A., & Wulfeck, B. (2001). Psycholinguistics: A cross-language perspective. Annual Review of Psychology, 52 (May 2000), 369–396.Google Scholar
Bitan, T., Burman, D. D., Chou, T.-L., Lu, D., Cone, N. E., Cao, F., Bigio, J. D., & Booth, J. R. (2007). The interaction between orthographic and phonological information in children: An fMRI study. Human Brain Mapping, 28 (9), 880–891.Google Scholar
Blumenfeld, H., & Marian, V. (2007). Constraints on parallel activation in bilingual spoken language processing: Examining proficiency and lexical status using eye-tracking. Language and Cognitive Processes, 22 (5), 633–660.CrossRefGoogle Scholar
Brysbaert, M., & New, B. (2009). Moving beyond Kucera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavioral Research Methods, 41 (4), 977–990.Google Scholar
Burgess, C., & Lund, K. (1997). Modelling parsing constraints with high-dimensional context space. Language & Cognitive Processes, 12 (2), 177–210.Google Scholar
Chater, N., & Christiansen, M. H. (2008). Computational models in psycholinguistics. In Sun, R. (ed.), Cambridge handbook of computational cognitive modeling, pp. 477–504. New York: Cambridge University Press.Google Scholar
Cleland, A. A., Gaskell, M. G., Quinlan, P. T., & Tamminen, J. (2006). Frequency effects in spoken and visual word recognition: Evidence from dual-task methodologies. Journal of Experimental Psychology: Human Perception and Performance, 32 (1), 104–119.Google Scholar
Costa, A., Caramazza, A., & Sebastián-Gallés, N. (2000). The cognate facilitation effect: Implications for models of lexical access. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26 (5), 1283–1296.Google Scholar
Costa, A., & Santesteban, M. (2004). Lexical access in bilingual speech production: Evidence from language switching in highly proficient bilinguals and L2 learners. Journal of Memory and Language, 50 (4), 491–511.Google Scholar
Costa, A., Santesteban, M., & Ivanova, I. (2006). How do highly proficient bilinguals control their lexicalization process? Inhibitory and language-specific selection mechanisms are both functional. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 1057–1074.Google Scholar
Cuetos, F., Glez-Nosti, M., Barbón, A., & Brysbaert, M. (2011). SUBTLEX-ESP: Spanish word frequencies based on film subtitles. Psicológica, 32, 133–143.Google Scholar
De Groot, A. M. B., & Kroll, J. (eds.) (1997). Tutorials in bilingualism: Psycholinguistic perspectives. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Dijkstra, T., Grainger, J., & van Heuven, W. J. B. (1999). Recognition of cognates and interlingual homographs: The neglected role of phonology. Journal of Memory and Language, 41, 496–518.Google Scholar
Dijkstra, T., & van Heuven, W. J. B. (1998). The BIA model and bilingual word recognition. In Grainger, J. & Jacobs, A. M. (eds.), Localist connectionist approaches to human cognition, pp. 189–225. Mahwah, NJ: Lawrence ErlbaumGoogle Scholar
Dijkstra, T., & van Heuven, W. J. B. (2002). The architecture of the bilingual word recognition system: From identification to decision. Bilingualism: Language and Cognition, 5 (3), 175–197.Google Scholar
Dong, Y., Gui, Sh., & MacWhinney, B. (2005). Shared and separate meanings in the bilingual mental lexicon. Bilingualism: Language and Cognition, 8 (3), 221–238.Google Scholar
Finkbeiner, M., Almeida, J., Janssen, N., & Caramazza, A. (2006). Lexical selection in bilingual speech production does not involve language suppression. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32 (5), 1075–1089.Google Scholar
Finkbeiner, M., Forster, K., Nicol, J., & Nakamura, K. (2004). The role of polysemy in masked semantic and translation priming. Journal of Memory and Language, 51 (1), 1–22.Google Scholar
FitzPatrick, I., & Indefrey, P. (2010). Lexical competition in nonnative speech comprehension. Journal of Cognitive Neuroscience, 22 (6), 1165–1178.Google Scholar
Forster, K. I. (1976). Accessing the mental lexicon. In Wales, R. J. & Walker, E. (eds.), New approaches to language mechanisms, pp. 257–287. Amsterdam: North-Holland.Google Scholar
Gentilucci, M., & Cattaneo, L. (2005). Automatic audiovisual integration in speech perception. Experimental Brain Research, 167 (1), 66–75.Google Scholar
Gollan, T. H., & Ferreira, V. S. (2009). Should I stay or should I switch? A cost-benefit analysis of voluntary language switching in young and aging bilinguals. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35 (3), 640–665.Google Scholar
Graham, R., & Belnap, K. (1986). The acquisition of lexical boundaries in English by native speakers of Spanish. International Review of Applied Linguistics, 24, 275–286.Google Scholar
Grainger, J., & Dijkstra, T. (1992). On the representation and use of language information in bilinguals. In Harris, R. J. (ed.), Cognitive processing in bilinguals (vol. 83), pp. 207–220. Amsterdam: Elsevier.Google Scholar
Grasemann, U., Kiran, S., Sandberg, C., & Miikkulainen, R. (2011). In Laaksonen, J. & Honkela, T. (eds.), Proceedings of WSOM11, 8th Workshop on Self-organizing Maps, pp. 207–217. Espoo: Springer Verlag.Google Scholar
Green, D. W. (1998). Mental control of the bilingual lexico-semantic system. Bilingualism: Language and Cognition, 1, 67–82.Google Scholar
Grosjean, F. (1988). Exploring the recognition of guest words in bilingual speech. Language and Cognitive Processes, 3, 233–274.Google Scholar
Grosjean, F. (1997). Processing mixed languages: Issues, findings and models. In De Groot, & Kroll, (eds.), pp. 225–254.Google Scholar
Hartsuiker, R. J., Pickering, M. J., & Veltkamp, E. (2004). Is syntax separate or shared between languages? Cross-linguistic syntactic priming in Spanish–English bilinguals. Psychological Science, 15 (6), 409–414.Google Scholar
Huettig, F., & Altmann, G. T. M. (2005). Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96, B23–32.Google Scholar
Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57, 460–482.Google Scholar
International Phonetic Association. (1999). Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge: Cambridge University Press.Google Scholar
Ju, M., & Luce, P. A. (2004). Falling on sensitive ears. Psychological Science, 15, 314–318.CrossRefGoogle ScholarPubMed
Jurgens, D., & Stevens, K. (2010). The S-Space package: An open source package for word space models. Proceedings of the ACL 2010 System Demonstrations, pp. 30–35. Association for Computational Linguistics.Google Scholar
Kanekama, Y., & Downs, D. (2009). Effects of speechreading and signal-to-noise ratio on understanding American English by American and Indian adults. Journal of the Acoustical Society of America, 126 (4), 2314.Google Scholar
Kaushanskaya, M., & Marian, V. (2007). Non-target language recognition and interference: Evidence from eye-tracking and picture naming. Language Learning, 57 (1), 119–163.CrossRefGoogle Scholar
Kaushanskaya, M, Yoo, J., & Marian, V. (2011). The effect of second-language experience on native-language processing. Vigo International Journal of Applied Linguistics, 8, 54–77.Google Scholar
Knoeferle, P., Crocker, M. W., Scheepers, C., & Pickering, M. J. (2005). The influence of the immediate visual context on incremental thematic role assignment: Evidence from eye-movements in depicted events. Cognition, 95 (1), 95–127.Google Scholar
Kohonen, T. (1995). Self-organizing maps. Berlin; Heidelberg; New York: Springer.Google Scholar
Kramer, A. F., & Donchin, E. (1987). Brain potentials as indices of orthographic and phonological interaction during word matching. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13 (1), 76–86.Google Scholar
Kroll, J. F., & De Groot, A. M. B. (1997). Lexical and conceptual memory in the bilingual: Mapping form to meaning in two languages. In De Groot, & Kroll, (eds.), pp. 169–199.Google Scholar
Kroll, J. F. & De Groot, A. M. B. (eds.) (2005). Handbook of bilingualism: Psycholinguistic approaches. New York: Oxford University Press.Google Scholar
Kroll, J. F., Dijkstra, T., Janssen, N., & Schriefers, H. (2000). Selecting the language in which to speak: Experiments on lexical access in bilingual production. Proceedings of the 41st Annual Meeting of the Psychonomic Society, New Orleans, LA.Google Scholar
Kroll, J. F., & Stewart, E. (1994). Category interference in translation and picture naming: Evidence for asymmetric connections between bilingual memory representations. Journal of Memory and Language, 33 (2), 149–174.Google Scholar
Lalor, E., & Kirsner, K. (2001). The representation of “false cognates” in the bilingual lexicon. Psychnomic Bulletin & Review, 8, 552–559.Google Scholar
Li, P. (1998). Mental control, language tags, and language nodes in bilingual lexical processing. Bilingualism: Language and Cognition, 1 (2), 92–93.CrossRefGoogle Scholar
Li, P., & Farkas, I. (2002). A self-organized connectionist model of bilingual processing. In Heredia, R. & Altarriba, J. (eds.), Bilingual sentence processing, pp. 59–85. Amsterdam: Elsevier.Google Scholar
Li, P., & MacWhinney, B. (2002). PatPho: A phonological pattern generator for neural networks. Behavior Research Methods, Instruments, and Computers, 34, 408–415Google Scholar
Li, P., Zhao, X., & MacWhinney, B. (2007). Dynamic Self-Organization and children's word learning. Cognitive Science, 31, 581–612.Google Scholar
Loebell, H., & Bock, K. (2003). Structural priming across languages. Linguistics, 41 (5), 791–824.CrossRefGoogle Scholar
Luce, P. A., & Pisoni, D. B. (1998). Recognizing spoken words: The neighborhood activation model. Ear & Hearing, 19, 1–36.Google Scholar
Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, 28 (2), 203–208.Google Scholar
Mani, N., & Plunkett, K. (2010). In the infant's mind's ear: Evidence for implicit naming in 18-month-olds. Psychological Science, 21 (7), 908–913.Google Scholar
Marchman, V. A., Fernald, A., & Hurtado, N. (2010). How vocabulary size in two languages relates to efficiency in spoken word recognition by young Spanish–English bilinguals. Journal of Child Language, 37 (4), 817–840.CrossRefGoogle ScholarPubMed
Marian, V. (2008). Bilingual research methods. In Altarriba, J. & Heredia, R. R. (eds.), An introduction to bilingualism: Principles and processes, pp. 13–38. Mahwah, NJ: Laurence Erlbaum.Google Scholar
Marian, V. (2009). Audio-visual integration during bilingual language processing. In Pavlenko, (ed.), pp. 52–78.CrossRefGoogle Scholar
Marian, V., & Spivey, M. (2003a). Bilingual and monolingual processing of competing lexical items. Applied Psycholinguistics, 24, 173–193.CrossRefGoogle Scholar
Marian, V., & Spivey, M. (2003b). Competing activation in bilingual language processing: Within- and between-language competition. Bilingualism: Language and Cognition, 6 (2), 97–115.Google Scholar
Marslen-Wilson, W. D. (1987). Functional parallelism in spoken word-recognition. Cognition, 25, 71–102.Google Scholar
McClelland, J. L., & Elman, J. L. (1986). The TRACE model of speech perception. Cognitive Psychology, 18, 1–86.Google Scholar
McClelland, J. L., Mirman, D., & Holt, L. L. (2006). Are there interactive processes in speech perception? Trends in Cognitive Sciences, 10, 363–369.Google Scholar
McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception, part 1: An account of basic findings. Psychological Review, 88, 375–407.Google Scholar
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.Google Scholar
McQueen, J. M., Norris, D., & Cutler, A. (2006). Are there really interactive processes in speech perception? Trends in Cognitive Sciences, 10 (12), 534.Google Scholar
Meuter, R. F. I., & Allport, A. (1999). Bilingual language switching in naming: Asymmetrical costs of language selection. Journal of Memory and Language, 40 (1), 25–40.Google Scholar
Meyer, A. S., Belke, E., Telling, A. L., & Humphreys, G. W. (2007). Early activation of objects names in visual search. Psychonomic Bulletin & Review, 14 (4), 710–716.Google Scholar
Miikkulainen, R. (1993). Subsymbolic natural language processing: An integrated model of scripts, lexicon, and memory. Cambridge, MA: MIT Press.Google Scholar
Miikkulainen, R. (1997). Dyslexic and category-specific aphasic impairments in a self-organizing feature map model of the lexicon. Brain and Language, 59, 334–366.Google Scholar
Miikkulainen, R., & Kiran, S. (2009). Modeling the bilingual lexicon of an individual subject. Lecture Notes in Computer Science, 5629, 191–199.Google Scholar
Morton, J. (1969). Interaction of information in word recognition. Psychological Review, 76 (2), 165–178.Google Scholar
Navarra, J., & Soto-Faraco, S. (2007). Hearing lips in a second language: Visual articulatory information enables the perception of second language sounds. Psychological Research, 71, 4–12.Google Scholar
Norris, D. (1994). Shortlist: A connectionist model of continuous speech recognition. Cognition, 52 (3), 189–234.Google Scholar
Norris, D., & McQueen, J. M. (2008). Shortlist B: A Bayesian model of continuous speech recognition. Psychological Review, 115 (2), 357–395.Google Scholar
Obleser, J., Leaver, A. M., VanMeter, J., & Rauschecker, J. P. (2010). Segregation of vowels and consonants in human auditory cortex: Evidence for distributed hierarchical organization. Frontiers in Psychology, 1 (December), 1–14.Google Scholar
Pavlenko, A. (ed.) (2009). The bilingual mental lexicon: Interdisciplinary approaches. Tonawanda, NY: Multilingual Matters.Google Scholar
Pavlenko, A., & Driagina, V. (2007). Russian emotion vocabulary in American learners’ narratives. Modern Language Journal, 91 (2), 213–234.Google Scholar
Rastle, K., & Brysbaert, M. (2006). Masked phonological priming effects in English: Are they real? Do they matter? Cognitive Psychology, 53 (2), 97–145.Google Scholar
Rastle, K., McCormick, S. F., Bayliss, L., & Davis, C. J. (2011). Orthography influences the perception and production of speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37 (6), 1588–1594.Google Scholar
Roelofs, A. (2003). Goal-referenced selection of verbal action: Modeling attentional control in the Stroop task. Psychological Review, 110 (1), 88–125.Google Scholar
Roelofs, A., & Verhoef, K. (2006). Modeling the control of phonological encoding in bilingual speakers. Bilingualism: Language and Cognition, 9, 167–176.Google Scholar
Samuel, A. G. (1996). Does lexical information influence the perceptual restoration of phonemes? Journal of Experimental Psychology: General, 125, 28–51.CrossRefGoogle Scholar
Samuel, A. G. (2001). Knowing a word affects the fundamental perception of the sounds within it. Psychological Science, 12, 348–351.Google Scholar
Salamoura, A., & Williams, J. N. (2007). Processing verb argument structure across languages: Evidence for shared representations in the bilingual lexicon. Applied Psycholinguistics, 28 (4), 627–660.Google Scholar
Sánchez-Casas, R., & García-Albea, J. E. (2005). The representation of cognate and noncognate words on bilingual memory: Can cognate status be characterized as a special kind of morphological relation? In Kroll, & De Groot, (eds.), pp. 226–250.Google Scholar
Schoonbaert, S., Duyck, W., Brysbaert, M., & Hartsuiker, R. J. (2009). Semantic and translation priming from a first language to a second and back: Making sense of the findings. Memory and Cognition, 37, 569–586.CrossRefGoogle ScholarPubMed
Schoonbaert, S., Hartsuiker, R. J., & Pickering, M. J. (2007). The representation of lexical and syntactic information in bilinguals: Evidence from syntactic priming. Journal of Memory and Language, 56 (2), 153–171.Google Scholar
Schwartz, A. I., Kroll, J. F., & Diaz, M. (2007). Reading words in Spanish and English: Mapping orthography to phonology in two languages. Language & Cognitive Processes, 22 (1), 106–129.Google Scholar
Shaoul, C., & Westbury, C. (2011). A USENET corpus (2005–2010). Edmonton, AB: University of Alberta. http://www.psych.ualberta.ca/~westburylab/downloads/usenetcorpus.download.html (retrieved April 15, 2011).Google Scholar
Shook, A., & Marian, V. (2012). Bimodal bilinguals co-activate both languages during spoken comprehension. Cognition, 124 (3), 314–324.Google Scholar
Spivey, M. J., Tanenhaus, M. K., Eberhard, K. M., & Sedivy, J. C. (2002). Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45 (4), 447–481.Google Scholar
Sumby, W. H., & Pollack, I. (1954). Visual contributions to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.Google Scholar
Tanenhaus, M. K., Spivey-Knowlton, M., Eberhard, K., & Sedivy, J. (1995). Integration of visual and linguistic information during spoken language comprehension. Science, 268, 1632–1634.Google Scholar
Thierry, G., & Wu, Y. J. (2007). Brain potentials reveal unconscious translation during foreign-language comprehension. Proceedings of the National Academy of Sciences, 104, 12530–12535.Google Scholar
Thomas, M. S. C., & van Heuven, W. J. B. (2005). Computational models of bilingual comprehension. In Kroll, & De Groot, (eds.), pp. 202–225.Google Scholar
van Heuven, W. J. B., Dijkstra, T., & Grainger, J. (1998). Orthographic neighborhood effects in bilingual word recognition. Journal of Memory and Language, 39, 458–483.CrossRefGoogle Scholar
Vitevitch, M. S., & Luce, P. A. (1998). When words compete: Levels of processing in spoken word perception. Psychological Science, 9, 325–329.Google Scholar
Vitevitch, M. S., & Luce, P. A. (1999). Probabilistic phonotactics and neighborhood activation in spoken word recognition. Journal of Memory and Language, 40 (3), 374–408.Google Scholar
Vroomen, J., van Linden, S., de Gelder, B., & Bertelson, P. (2007). Visual recalibration and selective adaptation in auditory-visual speech perception: Contrasting build-up courses. Neuropsychologia, 45, 572–577.Google Scholar
Yee, E., & Sedivy, J. (2006). Eye movements reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 1–14.Google Scholar
Yee, E., & Thompson-Schill, S. (2007). Does lexical activation flow from word meanings to word sounds during spoken word recognition? Poster presented at the 48th Annual Meeting of the Psychonomic Society, Long Beach, CA.Google Scholar
Zhao, X., & Li, P. (2007). Bilingual lexical representation in a self-organizing neural network. In McNamara, D. S. & Trafton, J. G. (eds.), Proceedings of the 29th Annual Cognitive Science Society, pp. 759–760. Austin, TX: Cognitive Science Society.Google Scholar
Zhao, X., & Li, P. (2010). Bilingual lexical interactions in an unsupervised neural network model. International Journal of Bilingual Education and Bilingualism, 13 (5), 505–524.Google Scholar
Ziegler, J. C., & Ferrand, L. (1998). Orthography shapes the perception of speech: The consistency effect in auditory word recognition. Psychonomic Bulletin & Review, 5 (4), 683–689.Google Scholar