The Knowledge Engineering Review

Articles

User-centered evaluation of adaptive and adaptable systems: a literature review

LEX VAN VELSENa1, THEA VAN DER GEESTa1, ROB KLAASSENa1 and MICHAËL STEEHOUDERa1

a1 Faculty of Behavioral Sciences, Institute for Behavioral Research, Department of Technical and Professional Communication, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands l.s.vanvelsen@utwente.nl, t.m.vandergeest@utwente.nl, r.klaassen@minaz.nl, m.f.steehouder@utwente.nl

Abstract

This literature review focuses on user-centered evaluation (UCE) studies of adaptive and adaptable systems. Usability, perceived usefulness and appropriateness of adaptation are the three most commonly assessed variables. Questionnaires appeared to be the most popular method, followed by interviews and data log analysis. The quality of most questionnaires was questionable, and the reporting of interviews and think-aloud protocols was found to be shallow. Furthermore, data logs need triangulation in order to be useful. The reports encountered lack empirical value. The article models the iterative design process for adaptive and adaptable systems, linked to the goals of UCE: supporting decisions, detecting problems and verifying quality. This model summarizes the variables to be assessed at each stage of the process and the relevant methods to assess them.