Journal of Functional Programming

Table of Contents - 1996 - Volume 6, Issue 04  

Articles

Benchmarking implementations of functional languages with ‘Pseudoknot’, a float-intensive benchmark

Pieter H. Hartela1, Marc Feeleya2, Martin Alta3, Lennart Augustssona4, Peter Baumanna5, Marcel Beemstera6, Emmanuel Chaillouxa7, Christine H. Flooda8, Wolfgang Grieskampa9, John H. G. Van Groningena10, Kevin Hammonda11, Bogumil Hausmana12, Melody Y. Ivorya13, Richard E. Jonesa14, Jasper Kampermana15, Peter Leea16, Xavier Leroya17, Rafael D. Linsa18, Sandra Loosemorea19, Niklas Röjemoa20, Manuel Serranoa21, Jean-Pierre Talpina22, Jon Thackraya23, Stephen Thomasa24, Pum Waltersa25, Pierre Weisa26 and Peter Wentwortha27

a1 Department of Computer Systems, University of Amsterdam, Kruislaan 403, 1098 SJ Amsterdam, The Netherlands (e-mail: pieter@fwi.uva.nl)

a2 Département d'informatique et r.o., Université de Montréal, succursale centre-ville, Montréal H3C 3J7, Canada (e-mail: feeley@iro.umontreal.ca)

a3 Informatik, Universität des Saarlandes, 66041 Saarbrücken 11, Germany (e-mail: alt@cs.uni-sb.de)

a4 Department of Computer Systems, Chalmers University of Technology, 412 96 Göteborg, Sweden (e-mail: augustss@cs.chalmers.se)

a5 Department of Computer Science, University of Zurich, Winterthurerstr. 190, 8057 Zurich, Switzerland (e-mail: baumann@ifi.unizh.ch)

a6 Department of Computer Systems, University of Amsterdam, Kruislaan 403, 1098 SJ Amsterdam, The Netherlands (e-mail: beemster@fwi.uva.nl)

a7 LIENS, URA 1327 du CNRS, École Normale Supérieure, 45 rue d'Ulm, 75230 PARIS Cédex 05, France (e-mail: Emmanuel.Chailloux@ens.fr)

a8 Laboratory for Computer Science, MIT, 545 Technology Square, Cambridge, MA 02139, USA (e-mail: chf@lcs.mit.edu)

a9 Berlin University of Technology, Franklinstr. 28-29, 10587 Berlin, Germany, (e-mail: wg@cs.tu-berlin.de)

a10 Faculty of Mathematics and Computer Science, Univ. of Nijmegen, Toernooiveld 1, 6525 ED Nijmegen, The Netherlands (e-mail: johnvg@cs.kun.nl)

a11 Department of Computing Science, Glasgow University, 17 Lilybank Gardens, Glasgow, GI2 8QQ, UK (e-mail: kh@dcs.glasgow.ac.uk)

a12 Computer Science Lab, Ellemtel Telecom Systems Labs, Box 1505, S-125 25 Älvsjö, Sweden (e-mail: bogdan@erix.ericsson.se)

a13 Computer Research Group, Institute for Scientific Computer Research, Lawrence Livermore National Laboratory, P.O. Box 808 L-419, Livermore, CA 94550, USA (e-mail: ivoryl@llnl.gov)

a14 Department of Computer Science, University of Kent at Canterbury, Canterbury, Kent, CT2 7NF, UK (e-mail: R.E.Jones@ukc.ac.uk)

a15 CWI, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands (e-mail: jasper@cwi.nl)

a16 Department of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue Pittsburgh, Pennsylvania 15213, USA (e-mail: petel@cs.cmu.edu)

a17 INRIA Rocquencourt, projet Cristal, B.P. 105, 78153 Le Chesnay, France (e-mail: Xavier.Leroy@inria.fr)

a18 Departamento de lnformática, Universidade Federal de Pernambuco, Recife, PE, Brazil (e-mail: rdl@di.ufpe.br)

a19 Department of Computer Science, Yale University, New Haven, CT, USA (e-mail: loosemore-sandra@cs.yale.edu)

a20 Department of Computer Systems, Chalmers University of Technology, 412 96 Göteborg, Sweden (e-mail: rojemo@cs.chalmers.se)

a21 INRIA Rocquencourt, projet Icsla, B.P. 105, 78153 Le Chesnay, France (e-mail: Manuel.Serrano@inria.fr)

a22 European Computer-Industry Research Centre, Arabella Straβe 17, D-81925 Munich, Germany (e-mail: jp@ecrc.de)

a23 Harlequin Ltd, Barrington Hall, Barrington, Cambridge CB2 5RG, UK (e-mail: jont@harlequin.co.uk)

a24 Department of Computer Science, University of Nottingham, Nottingham, NG7 2RD, UK (e-mail: spt@cs.nott.ac.uk)

a25 CWI, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands (e-mail: pum@cwi.nl)

a26 INRIA Rocquencourt, projet Cristal, B.P. 105, 78153 Le Chesnay, France (e-mail: Pierre.Weis@inria.fr)

a27 Department of Computer Science, Rhodes University, Grahamstown 6140, South Africa (e-mail: cspw@cs.ru.ac.za)

Abstract

Over 25 implementations of different functional languages are benchmarked using the same program, a floating-point intensive application taken from molecular biology. The principal aspects studied are compile time and execution time for the various implementations that were benchmarked. An important consideration is how the program can be modified and tuned to obtain maximal performance on each language implementation. With few exceptions, the compilers take a significant amount of time to compile this program, though most compilers were faster than the then current GNU C compiler (GCC version 2.5.8). Compilers that generate C or Lisp are often slower than those that generate native code directly: the cost of compiling the intermediate form is normally a large fraction of the total compilation time. There is no clear distinction between the runtime performance of eager and lazy implementations when appropriate annotations are used: lazy implementations have clearly come of age when it comes to implementing largely strict applications, such as the Pseudoknot program. The speed of C can be approached by some implementations, but to achieve this performance, special measures such as strictness annotations are required by non-strict implementations. The benchmark results have to be interpreted with care. Firstly, a benchmark based on a single program cannot cover a wide spectrum of ‘typical’ applications. Secondly, the compilers vary in the kind and level of optimisations offered, so the effort required to obtain an optimal version of the program is similarly varied.