Hostname: page-component-76fb5796d-vvkck Total loading time: 0 Render date: 2024-04-25T16:29:50.622Z Has data issue: false hasContentIssue false
  • English
  • Français

Audiovisual Harmony: The realtime audiovisualisation1 of a single data source in Construction in Zhuangzi

Published online by Cambridge University Press:  19 July 2012

Ryo Ikeshiro
Ryo Ikeshiro*
Affiliation:
Department of Music and Department of Computing, Goldsmiths, University of London, New Cross, London, SE14 6NW, UK
*
E-mail: info@ryoikeshiro.com
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper explores the context and technical and aesthetic considerations behind the author's generative and improvisational audiovisual work, Construction in Zhuangzi (2011), and in particular the approach of ‘audiovisualising’ the same source of data and its validation, and its possibilities as an artistic practice. First, the origins of integrated audiovisual art in the output of John Whitney are explored. Then, metaphors based on musical textures are used to describe different approaches to the audiovisual medium. Research into perception and auditory displays are next used to justify the simultaneous representation of the same data in both the audio and the video. The aesthetic potential of this practice is then corroborated using Michel Chion's theory of sound in cinema. In concluding, its possibilities for providing an appropriate form and aesthetic approach to the audiovisual material are discussed.

Type
Articles
Information
Organised Sound , Volume 17 , Issue 2: Composing Motion: A visual music retrospective , August 2012 , pp. 148 - 155
Copyright
Copyright © Cambridge University Press 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

1

The term is used throughout the paper to indicate simultaneous sonification and visualisation, sonification consisting of both direct audification i.e. the translation of data into the audible range, and parameter mappings related to the production of audio.

References

Abbado, A. 1988. Perceptual Correspondences of Abstract Animation and Synthetic Sound. Leonardo Supplemental Issue 1: 3–5.Google Scholar
Alves, B. 2005. Digital Harmony of Sound and Light. Computer Music Journal 29(4): 45–54.CrossRefGoogle Scholar
Bertelson, P., Radeau, M. 1981. Cross-modal Bias and Perceptual Fusion with Auditory-visual Spatial Discordance. Perception & Psychophyics 29(6): 578–584.CrossRefGoogle ScholarPubMed
Bouchara, T., Katz, B.F.G., Jacquemin, C., Guastavino, C. 2010. Audio-visual Renderings for Multimedia Navigation. Proceeding of the 16th International Conference on Auditory Display. Washington, DC: ICAD.Google Scholar
Cádiz, R.F. 2004. A Fuzzy Logic Model for Compositional Approaches to Audio-Visual Media. Proceedings of the International Computer Music Conference. Miami: ICMA.Google Scholar
Chion, M. 1994. Audio-Vision: Sound On Screen. Foreword by W. Murch. New York: Columbia University Press.Google Scholar
Coulter, J. 2010. Electroacoustic Music with Moving Images: The Art of Media Pairing. Organised Sound 15(1): 26–34.CrossRefGoogle Scholar
CVM (Center for Visual Music). 2012. Research: Errata. http://www.centerforvisualmusic.org/ResearchErrata.html. Accessed on 19 April 2012.Google Scholar
DeLanda, M. 2002. Intensive Science and Virtual Philosophy. London and New York: Continuum.Google Scholar
Deleuze, G., Guattari, F. 1994. What is Philosophy?. London and New York: Verso.Google Scholar
Di Scipio, A. 2001. Iterated Nonlinear Functions as a Sound-Generating Engine. Leonardo 34(3): 249–254.CrossRefGoogle Scholar
Dombois, F. 2002. Auditory Seismology: On Free Oscillations, Focal Mechanisms, Explosions, and Synthetic Seismograms. Proceedings of the 8th International Conference on Auditory Display. Kyoto: ICAD, pp. 27–30.Google Scholar
Edmonds, E. 2003. Logics for Constructing Generative Art Systems. Digital Creativity 14(1): 23–28.CrossRefGoogle Scholar
Faulkner, M./D-Fuse (ed.) 2006. VJ: Audio-visual Art and VJ Culture. London: Laurence King.Google Scholar
Ghirardelli, G.T., Scharine, A.A. 2009. Auditory-visual Interactions. In C.E. Rash, M.B. Russo, T.R. Letowski and E.T. Schmeisser (eds) Helmet-Mounted Displays: Sensation, Perception and Cognition Issues. Fort Rucker, AL: US Army Aeromedical Research Laboratory.Google Scholar
Grierson, M. 2005. Audiovisual Composition. PhD thesis, University of Kent.Google Scholar
Gröhn, M., Lokki, T., Takala, T. 2003. Comparison of Auditor, Visual, and Audio-Visual Navigation in a 3D Space. Proceedings of the 2003 International Conference on Auditory Display. Boston: ICAD.Google Scholar
Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J.H., Miner, N., Neuhoff, J. 1997. Sonification Report: Status of the Field and Research Agenda. Prepared for the NSF by members of ICAD.Google Scholar
Levin, G. 2000. Painterly Interfaces for Audiovisual Performance. MSc thesis, Massachusetts Institute of Technology.Google Scholar
Lorenz, E.N. 1963. Deterministic Nonperiodic Flow. Journal of the Atmospheric Sciences 20: 130–141.2.0.CO;2>CrossRefGoogle Scholar
Manovich, L. 2001. The Language of New Media. Cambridge, MA: The MIT Press.Google Scholar
McGurk, H., MacDonald, J. 1976. Hearing Lips and Seeing Voices. Nature 264: 746–748.CrossRefGoogle ScholarPubMed
Peacock, K. 1988. Instruments to Perform Color-music: Two Centuries of Technological Experimentation. Leonardo 21(4): 397–406.CrossRefGoogle Scholar
Petržela, J., Hanus, S. 2005. Experimental Observations of Strange Attractors in the Modified Lorenz System. Proceedings of the 15th International Conference Radioelektronika. Brno: Radioelektronika.Google Scholar
Radeau, M., Bertelson, P. 1977. Adaptation to Auditory–visual Discordance and Ventriloquism in Semirealistic Situations. Perception and Psychophysics 22: 137–146.CrossRefGoogle Scholar
Ramachandran, V.S., Hubbard, E.M. 2003. Hearing Colors, Tasting Shapes. Scientific American 288(5): 53–59.CrossRefGoogle ScholarPubMed
Ritter, D. 1993. Interactive Video as a Way of Life. Music Works 56: 48–54.Google Scholar
Roads, C. 1996. The Computer Music Tutorial. Cambridge, MA: The MIT Press.Google Scholar
Rudi, J. 1998. Computer Music Animations. Organised Sound 3(3): 193–198.CrossRefGoogle Scholar
Saue, S., Fjeld, O. 1997. A Platform for Audiovisual Seismic Interpretation. Proceedings of the 4th International Conference on Auditory Display. Palo Alto, CA: ICAD.Google Scholar
Smith, D.W. 2003. Mathematics and the Theory of Multiplicities: Badiou and Deleuze Revisited. Southern Journal of Philosophy 41(3): 411–449.CrossRefGoogle Scholar
Talsma, D., Woldorff, M.G. 2005. Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity. Journal of Cognitive Neuroscience 17(7): 1098–1114.CrossRefGoogle ScholarPubMed
Vroomen, J., de Gelder, B. 2004. Perceptual Effects of Cross-modal Stimulation: Ventriloquism and the Freezing Phenomenon. In G. Calvert, C. Spence and B.E. Stein (eds) The Handbook of Multisensory Processes. Cambridge, MA: The MIT Press.Google Scholar
Walker, B.N., Kramer, G. 2004. Ecological Psychoacoustics and Auditory Displays: Hearing, Grouping, and Meaning Making. In J.G. Neuhoff (ed.) Ecological Psychoacoustics. San Diego, CA: Elsevier Academic Press, 150–175.Google Scholar
Whitney, J. 1980. Digital Harmony: On the Complementarity of Music and Visual Art. Peterborough, NH: Byte Books.Google Scholar
Whitney, J. 1991. Fifty Years of Composing Computer Music and Graphics: How Time's New Solid-state Tractability Has Changed Audio-Visual Perspectives. Leonardo 24(5): 597–599.CrossRefGoogle Scholar
Whitney, J. 1994. To Paint on Water: The Audiovisual Duet of Complementarity. Computer Music Journal 18(3): 45–52.CrossRefGoogle Scholar

Ikeshiro Supplementary Movie

Movie_1

Download Ikeshiro Supplementary Movie(Video)
Video 8.3 MB

Ikeshiro Supplementary Movie

Movie_2

Download Ikeshiro Supplementary Movie(Video)
Video 2 MB

Ikeshiro Supplementary Movie

Movie_3

Download Ikeshiro Supplementary Movie(Video)
Video 6 MB

Ikeshiro Supplementary Movie

Movie_4

Download Ikeshiro Supplementary Movie(Video)
Video 70.8 MB

Ikeshiro Supplementary Movie

Movie_5

Download Ikeshiro Supplementary Movie(Video)
Video 8.8 MB

Ikeshiro Supplementary Movie

Movie_6

Download Ikeshiro Supplementary Movie(Video)
Video 4.4 MB

Ikeshiro Supplementary Movie

Movie_7

Download Ikeshiro Supplementary Movie(Video)
Video 36.6 MB