Hostname: page-component-7c8c6479df-5xszh Total loading time: 0 Render date: 2024-03-28T18:52:05.337Z Has data issue: false hasContentIssue false

Pragmatic design in randomized controlled trials

Published online by Cambridge University Press:  02 June 2014

M. Purgato*
Affiliation:
WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, Department of Public Health and Community Medicine, Section of Psychiatry, University of Verona, Verona, Italy
C. Barbui
Affiliation:
WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, Department of Public Health and Community Medicine, Section of Psychiatry, University of Verona, Verona, Italy
S. Stroup
Affiliation:
Psychiatric Institute, Columbia University, New York, NY, USA
C. Adams
Affiliation:
Division of Psychiatry, University of Nottingham, Nottingham, UK
*
*Address for correspondence: M. Purgato, WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, Department of Public Health and Community Medicine, Section of Psychiatry, University of Verona, Verona, Italy. (Email: marianna.purgato@univr.it)
Rights & Permissions [Opens in a new window]

Abstract

At more than 10 years after the paper by Hotopf and colleagues regarding pragmatic trials in psychiatry, the field has evolved and is evolving further. There have been many developments in our understanding of what pragmatism really means, and excellent examples of truly pragmatic trials in psychiatry are currently available. Funders have helped encourage more emphasis on the need for such studies, but ‘local’ and trans-national regulations could help more. Consumers of the evidence should have a greater voice in generating the research agenda and, as this happens, the questions generated are more likely to be answered by a pragmatic approach to trials.

Type
Editorial
Copyright
Copyright © Cambridge University Press 2014 

Introduction

It has been over 10 years since the paper of Hotopf et al. (Reference Hotopf, Churchill and Lewis1999) on pragmatic trials. This understandable call was partly a response to the increasing realization that even well-conducted randomized controlled trials did not necessarily address the needs of everyday care. Most trials involved participant groups so selected that they would be rare in routine care, and evaluated treatments administered so rigidly as to further remove the practice from reality and, finally, measured outcomes that were of questionable importance to clinicians, policymakers or recipients of care. The article by Hotopf et al. (Reference Hotopf, Churchill and Lewis1999) highlighted some of the benefits of pragmatic trials but with few examples to point to at that time. Most lessons had been learnt from colleagues in cancer, heart disease and perinatal medicine. Since that time, however, there have been many developments regarding our understanding of what pragmatism really means in trials and in how to conduct these types of studies in mental health. Further papers followed that highlighted a developing understanding of these types of studies within mental health care evaluation with increasing numbers of examples to cite (March et al. Reference March, Silva, Compton, Shapiro, Califf and Krishnan2005; Stroup, Reference Stroup2011).

Despite this, ‘pragmatic’ remains a much misunderstood concept within trials and is often used as a broad, fashionable and, often, unhelpful label. Over the last years it became apparent that the concept of pragmatism in randomized trials was not binary (Stroup & Geddes, Reference Stroup and Geddes2008). It has been recognized that considering randomized controlled trials as purely explanatory (studies that focused on whether, in ideal circumstances, a treatment would have any effect) or pragmatic (a study which investigated whether the treatment, given in real-world circumstances, really did have clinically meaningful effects) was too simplistic. The concept of explanatory and pragmatic trials being at opposite poles was dismissed in favour of the idea that there was a continuum between maximally pragmatic or fully explanatory.

Recently there has been a genuine advance in our understanding of pragmatism within trials (Thorpe et al. Reference Thorpe, Zwarenstein, Oxman, Treweek, Furberg, Altman, Tunis, Bergel, Harvey, Magid and Chalkidou2009). The authors defined 10 domains encapsulating explanatory and pragmatic approaches and covering the most important aspects of a trial design (Table 1).

Table 1. The 10 domains relevant to explanatory and pragmatic trials a

Thorpe et al. (Reference Thorpe, Zwarenstein, Oxman, Treweek, Furberg, Altman, Tunis, Bergel, Harvey, Magid and Chalkidou2009) went on to define and explain each of these domains in detail and have left us with an elegant tool by which pragmatism could be estimated. These attractive diagrams (Fig. 1) can be created for those designing trials or assessing trials at the protocol stage of the funding application process to give some impression of applicability to clinical circumstances. Estimates are made regarding each parameter and this can result in ‘open’ or ‘closed’ diagrams illustrating how pragmatic or explanatory a trial is likely to be (Fig. 2).

Fig. 1. The blank Pragmatic–Explanatory Continuum Indicator Summary (PRECIS) ‘wheel’.

Fig. 2. Two examples of trials.

Understandably, this has triggered papers outlining specific tools for both trials and studies within systematic reviews (Tosh et al. Reference Tosh, Soares-Weiser and Adams2011). These tools, with various degrees of reliability, can produce numerical estimates of the applicability of trials to the real world (Table 2).

Table 2. Scoring trial protocols estimating degrees of pragmatism a

ACHIEVE, Achieving healthy lifestyles in psychiatric rehabilitation; ERP, enhanced relapse prevention; DYD, Down Your Drink; PTSD – Yoga, Post-traumatic stress disorder – Yoga; CATIE, Clinical Antipsychotic Trials of Intervention Effectiveness; FIAT, Financial Incentives (for) Adherence Trial; TREC-SAVE, Tranquilizǎcção Rápida - Ensaio Clínic: Segurança no manejo da Agressão: Viabilidade e Ética na contenção e isolamento.

a From Tosh et al. (Reference Tosh, Soares-Weiser and Adams2011) and corrected to remove reference errors in the source table.

Mental health now has a series of trials to illustrate, to greater or lesser degrees, a more pragmatic approach. Previously the speciality had no choice but to employ explanatory trials in policy making. Currently, however, with increases in skills and experience, decision makers begin to have options. The last decade has seen a series of trials that falls much more into a pragmatic category. For example, Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE; Lieberman et al. Reference Lieberman, Stroup, McEvoy, Swartz, Rosenheck, Perkins, Keefe, Davis, Davis, Lebowitz, Severe and Hsiao2005) and Cost Utility of the Latest Antipsychotic Drugs in Schizophrenia Study (CUtLASS; Jones et al. Reference Jones, Barnes, Davies, Dunn, Lloyd, Hayhurst, Murray, Markwick and Lewis2006) are important ground-breaking studies with greater levels of pragmatism than had previously been seen in such large mental health trials. Other randomized studies such as BALANCE (Bipolar Affective disorder: Lithium/ANticonvulsant Evaluation; Geddes et al. Reference Geddes, Rendell and Goodwin2002) and the series of Tranquilizǎcção Rápida – Ensaio Clínic (TREC; Alexander et al. Reference Alexander, Tharyan, Adams, John, Mol and Philip2004; Huf et al. Reference Huf, Coutinho and Adams2012) trials further developed techniques in pragmatism within mental health. All juggle the differing needs of researcher, clinician, policymaker and service user within a trial. In these studies the concept of pragmatism has been declined in different ways, but some key common characteristics include the following. First, all these studies investigated already marketed medicines, with the aim of answering comparative effectiveness. Second, the choice of control group interventions was mainly based on prevailing clinical practice. Third, patient populations were included minimizing exclusion criteria and being focused more on clinical presentation than on formal diagnoses. Fourth, the choice of outcome measures was based on pragmatic reasoning around what really outcome means for practising doctors. Finally, the sample size was calculated to show superiority, which implies that value is attributed to treatments that are associated with some degree of advantage over standard ones.

Clearly, critical issues may arise from such study designs. For example, the concept of ‘real life’ cannot be unambiguously defined, and therefore the results of pragmatic trials carried out in European countries might be hardly applicable to a very different system of care, such as, for example, a very low-income system of care. The same criticism may be applied to the choice of control treatments in pragmatic trials, as prevailing clinical practice may highly differ and therefore what may be relevant is a specific system of care may not be similarly relevant in others. Internal validity might be another critical aspect, as in some circumstances it is difficult to identify the ‘active ingredients’ of effective interventions assessed under a highly pragmatic design.

Progress has also been made in relation to reporting of trials. The CONSORT (Consolidated Standards of Reporting Trials) statement – a tool intended to improve quality of reporting and minimize risk of bias – has been extended to take into account pragmatism of trials (Zwarenstein et al. Reference Zwarenstein, Treweek, Gagnier, Altman, Tunis, Haynes, Oxman and Moher2008). Additional text was added specifically for eight items, with detailed information on background, participants, interventions, outcomes, sample sizes, blinding, participant flow and generalizability of findings. These extensions represent an advance towards transparent and complete reporting of trials in general, and, in relation to the pragmatism of a trial, a guide for clinicians and policymakers for judging applicability to everyday clinical practice (Zwarenstein et al. Reference Zwarenstein, Treweek, Gagnier, Altman, Tunis, Haynes, Oxman and Moher2008). This progress came from a change in culture, not least of which was in psychiatry, where farsighted researchers and clinicians anticipated the broad and hitherto often overlooked needs of the many stakeholders for trial data (Tansella et al. Reference Tansella, Thornicroft, Barbui, Cipriani and Saraceno2006).

The next 10 years

There is more to do. Legislation could dramatically change the landscape for pragmatic trials.

The regulatory authorities

The issue of pragmatic versus explanatory is a reflection of the aim or objectives of the trial. Pragmatic trials aim to provide answers to real-world questions perceived to be relevant to patients, clinicians and policymakers. Common to all such trials are measures of patient-valued outcomes. Regulatory trials do not necessarily aim to answer these types of questions. Regulatory trials aim to obtain regulatory approval. Currently, phase III trials are often carried out by industry and are highly explanatory, while phase IV trials may more often be carried out by groups of physicians who have a clinical dilemma and are more pragmatic (Barbui & Bighelli, Reference Barbui and Bighelli2013b ).

The Food and Drug Administration in the USA and the European Medicines Agency (EMA) are regulatory bodies that establish rules to get new medicines approved and marketed. The current explanatory standard of phase III studies, therefore, is a consequence of current rules and requirements issued by these agencies. Changing regulatory requirements could greatly encourage a more real-world and pragmatic set of designs within phase III studies (Wood, Reference Wood2006). For example, in Europe new drugs can be evaluated with no comparison with active alternative treatments. As a consequence, phase III studies compare new drugs with placebo to make these new drugs eligible for registration – often not the question troubling the ‘real world’ of clinical care in which there may well be viable alternative drugs. If comparisons with other drugs are made within phase III studies, these usually rely on demonstrating therapeutic ‘non-inferiority’ – in agreement with current EMA requirements. If the scientific community agrees that active–control superiority is desirable, clinical trials should be designed and powered to generate the evidence and European pharmaceutical legislation should then be induced to incorporate this requirement, at least in addition to placebo-controlled trials (Garattini & Chalmers, Reference Garattini and Chalmers2009). It has recently been suggested that regulatory authorities may require at least one two-arm head-to-head pragmatic trial in order to demonstrate superiority of the investigational product over an active comparator (Barbui & Bighelli, Reference Barbui and Bighelli2013a ).

Conduct of studies

Increasingly, government funders of studies have become aware that the trials they have supported did not have the population-wide impact that they had intended (Duijnhoven et al. Reference Duijnhoven, Straus, Raine, de Boer, Hoes and De Bruin2013). Explanatory studies are less likely to achieve wide impact for the general population because, by definition, those eligible for the explanatory study are a highly specific group of people from which most of the general population is excluded. As a result taxpayers’ money has funded studies that help far fewer taxpayers than had been originally intended. Broad-brush stipulation from funders – that real-world applicability is required in trial design (NHS National Institute for Health Research, 2013)  – and then authoritative governance to ensure that funding panels rigorously adhere to this stipulation can greatly help shift research culture and tradition. Additional legislation can help more. A case illustration helps. In Italy, the Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico (GISSI) trial was a highly pragmatic study investigating intravenous streptokinase in early acute myocardial infarction (Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico, 1986). This enrolled 11806 patients in 176 coronary care units. Subsequent to the first highly influential paper of 1986, debate focused around the need for more similar studies. In 2004 a Ministerial Decree was issued recognizing the public health value of the original study and the need of establishing rules to help implement pragmatic independent phase IV clinical trials (Tognoni & Franzosi, Reference Tognoni and Franzosi2005). The Decree states that if a set of conditions is met (study coordinating centre is independent from drug companies, study results can be disseminated autonomously, there is no personal financial interest in studying the drugs included in the trial, study drugs are already in the market and are studied within the licensed indications), then the Italian health system supports the conduct of the trial in three ways. First, drug costs are paid by the Italian health system; (2) there are no fees for submitting the study protocol to the local ethics committees; (3) continuing medical education credits are provided to local investigators. The impact of this Decree on the development of pragmatic experimental studies has not been assessed yet, but from a cultural and theoretical viewpoint it has already been seen as a major advance in Italy.

Conclusions

When Hotopf et al. (Reference Hotopf, Churchill and Lewis1999) wrote about pragmatic trials in psychiatry their tone was almost wistful as regards other more advanced specialities. Things have changed and are changing further. We now have excellent examples of truly pragmatic trials in mental health. Funders have helped encourage more emphasis on the need for such studies but ‘local’ and trans-national regulations could help more. Consumers of the evidence should have a greater voice in generating the research agenda and, as this happens (Lloyd & White, Reference Lloyd and White2011), the questions generated are, assuredly, more likely to be answered by a pragmatic approach to trials.

If we do not devalue the ‘pragmatic’ label through misuse (Sackett, Reference Sackett2013), the next decade will see a burgeoning of mental health trials with a much more emphasis on showing if treatments really work in everyday care.

Acknowledgements

This paper received no specific grant from any funding agency.

Declaration of Interest

None.

References

Alexander, J, Tharyan, P, Adams, C, John, T, Mol, C, Philip, J (2004). Rapid tranquillisation of violent or agitated patients in a psychiatric emergency setting. Pragmatic randomised trial of intramuscular lorazepam v. haloperidol plus promethazine. British Journal of Psychiatry 185, 6369.Google Scholar
Barbui, C, Bighelli, I (2013 a). A new approach to psychiatric drug approval in Europe. PLoS Medicine 10, e1001530.Google Scholar
Barbui, C, Bighelli, I (2013 b). Regulatory science in Europe: the case of schizophrenia trials. Lancet 382, 12341235.CrossRefGoogle ScholarPubMed
Casagrande, SS, Jerome, GJ, Dalcin, AT, et al. (2010) Randomized trial of achieving healthy lifestyles in psychiatric rehabilitation: the ACHIEVE trial. BMC Psychiatry 10, 108.Google Scholar
Chalmers, I (1993). The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences 703, 156–65.CrossRefGoogle ScholarPubMed
Duijnhoven, RG, Straus, SM, Raine, JM, de Boer, A, Hoes, AW, De Bruin, ML (2013). Number of patients studied prior to approval of new medicines: a database analysis. PLoS Medicine 10, e1001407.CrossRefGoogle ScholarPubMed
Garattini, S, Chalmers, I (2009). Patients and the public deserve big changes in evaluation of drugs. British Medical Journal 338, b1025.Google Scholar
Geddes, JR, Rendell, JM, Goodwin, GM (2002). BALANCE: a large simple trial of maintenance treatment for bipolar disorder. World Psychiatry 1, 4851.Google Scholar
Gold, C, Rolvsjord, R, Aaro, LE, Aarre, T, Tjemsland, L, Stige, B (2005). Resource oriented music therapy for psychiatric patients with low therapy motivation: protocol for a randomised controlled trial. BMC Psychiatry 5, 39.CrossRefGoogle ScholarPubMed
Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico (GISSI) (1986). Effectiveness of intravenous thrombolytic treatment in acute myocardial infarction. Lancet i, 397402.Google Scholar
Hotopf, M, Churchill, R, Lewis, G (1999). Pragmatic randomised controlled trials in psychiatry. British Journal of Psychiatry 175, 217223.Google Scholar
Huf, G, Coutinho, ES, Adams, CE; TREC-SAVE Collaborative Group (2012). Physical restraints versus seclusion room for management of people with acute aggression or agitation due to psychotic illness (TREC-SAVE): a randomized trial. Psychological Medicine 42, 22652273.Google Scholar
Jones, PB, Barnes, TR, Davies, L, Dunn, G, Lloyd, H, Hayhurst, KP, Murray, RM, Markwick, A, Lewis, SW (2006). Randomized controlled trial of the effect on Quality of Life of second- vs first-generation antipsychotic drugs in schizophrenia: Cost Utility of the Latest Antipsychotic Drugs in Schizophrenia Study (CUtLASS 1). Archives of General Psychiatry 63, 10791087.Google Scholar
Lieberman, JA, Stroup, TS, McEvoy, JP, Swartz, MS, Rosenheck, RA, Perkins, DO, Keefe, RS, Davis, SM, Davis, CE, Lebowitz, BD, Severe, J, Hsiao, JK (2005). Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. New England Journal of Medicine 353, 12091223.CrossRefGoogle ScholarPubMed
Lloyd, K, White, J (2011). Democratizing clinical research. Nature 474, 277278.CrossRefGoogle ScholarPubMed
Lobban, F, Gamble, C, Kinderman, P, Taylor, L, Chandler, C, Tyler, E, Peters, S, Pontin, E, Sellwood, W, Morriss, RK (2007). Enhanced relapse prevention for bipolar disorder – ERP trial. A cluster randomised controlled trial to assess the feasibility of training care coordinators to offer enhanced relapse prevention for bipolar disorder. BMC Psychiatry 7, 6.Google Scholar
March, JS, Silva, SG, Compton, S, Shapiro, M, Califf, R, Krishnan, R (2005). The case for practical clinical trials in psychiatry. American Journal of Psychiatry 162, 836846.CrossRefGoogle ScholarPubMed
Morriss, R, Marttunnen, S, Garland, A, Nixon, N, McDonald, R, Sweeney, T, Flambert, H, Fox, R, Kaylor-Hughes, C, James, M, Yang, M (2010). Randomised controlled trial of the clinical and cost effectiveness of a specialist team for managing refractory unipolar depressive disorder. BMC Psychiatry 10, 100.Google Scholar
Murray, E, McCambridge, J, Khadjesari, Z, White, IR, Thompson, SG, Godfrey, C, Linke, S, Wallace, P (2007). The DYD-RCT protocol: an on-line randomised controlled trial of an interactive computer-based intervention compared with a standard information website to reduce alcohol consumption among hazardous drinkers. BMC Public Health 7, 306.CrossRefGoogle ScholarPubMed
NHS National Institute for Health Research (2013). HTA programme remit. The Health Technology Assessment Programme (http://www.hta.ac.uk/funding/remit.shtml). Accessed 30 October 2013.Google Scholar
Priebe, S, Burton, A, Ashby, D, Ashcroft, R, Burns, T, David, A, Eldridge, S, Firn, M, Knapp, M, McCabe, R (2009). Financial incentives to improve adherence to anti-psychotic maintenance medication in non-adherent patients – a cluster randomised controlled trial (FIAT). BMC Psychiatry 9, 61.CrossRefGoogle ScholarPubMed
Sackett, DL (2013). Clinician-trialist rounds: 17. Mind your explanatory and pragmatic attitudes! Part 2: How? Clinical Trials 10, 633636.Google Scholar
Stroup, TS (2011). What can large simple trials do for psychiatry? American Journal of Psychiatry 168, 117119.Google Scholar
Stroup, TS, Geddes, JR (2008). Randomized controlled trials for schizophrenia: study designs targeted to distinct goals. Schizophrenia Bulletin 34, 266274.Google Scholar
Stroup, TS, McEvoy, JP, Swartz, MS, Byerly, MJ, Glick, ID, Canive, JM, McGee, MF, Simpson, GM, Stevens, MC, Lieberman, JA (2003). The National Institute of Mental Health Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) project: schizophrenia trial design and protocol development. Schizophrenia Bulletin 29, 1531.Google Scholar
Tansella, M, Thornicroft, G, Barbui, C, Cipriani, A, Saraceno, B (2006). Seven criteria for improving effectiveness trials in psychiatry. Psychological Medicine 36, 711720.Google Scholar
Telles, S, Singh, N, Joshi, M, Balkrishna, A (2010). Post traumatic stress symptoms and heart rate variability in Bihar flood survivors following yoga: a randomized controlled study. BMC Psychiatry 10, 18.Google Scholar
Thorpe, KE, Zwarenstein, M, Oxman, AD, Treweek, S, Furberg, CD, Altman, DG, Tunis, S, Bergel, E, Harvey, I, Magid, DJ, Chalkidou, K (2009). A Pragmatic–Explanatory Continuum Indicator Summary (PRECIS): a tool to help trial designers. Canadian Medical Association Journal 180, E47E57.Google Scholar
Tognoni, G, Franzosi, MG (2005). [Experimenting in Italy: from GISSI to the Ministerial Decree]. Recenti Progressi in Medicina 96, 9294.Google Scholar
Tosh, G, Soares-Weiser, K, Adams, CE (2011). Pragmatic vs explanatory trials: the Pragmascope tool to help measure differences in protocols of mental health randomized controlled trials. Dialogues in Clinical Neuroscience 13, 209215.CrossRefGoogle ScholarPubMed
Wood, AJ (2006). A proposal for radical changes in the drug-approval process. New England Journal of Medicine 355, 618623.Google Scholar
Zwarenstein, M, Treweek, S, Gagnier, JJ, Altman, DG, Tunis, S, Haynes, B, Oxman, AD, Moher, D (2008). Improving the reporting of pragmatic trials: an extension of the CONSORT statement. British Medical Journal 337, a2390.Google Scholar
Figure 0

Table 1. The 10 domains relevant to explanatory and pragmatic trialsa

Figure 1

Fig. 1. The blank Pragmatic–Explanatory Continuum Indicator Summary (PRECIS) ‘wheel’.

Figure 2

Fig. 2. Two examples of trials.

Figure 3

Table 2. Scoring trial protocols estimating degrees of pragmatisma