INTRODUCTION

Traditionally, randomized controlled trials (RCTs) in controlled settings have been considered the gold standard for studying the efficacy of health interventions.¹⁾ It is the backbone of clinical research because randomization in well-defined and controlled settings can improve the internal validity and reduce bias, but these settings tend to overestimate the benefits and underestimate the risks.²⁾ Recently, there has been great interest in conducting post-launch observational studies that increase the external validity using real-world data (e.g., claims data and electronic health records [EHRs]) including a large population.¹⁾³⁾ However, several challenges have been reported when conducting these studies, including bias (e.g., selection bias), inaccuracy of diagnostic codes, and incomparability between patient groups.⁴⁾⁵⁾ In terms of real-world evidence (RWE) delivery, pragmatic clinical trials (PCTs) are proposed as a method to provide evidence in a relatively shorter time to solve problems faced by multiple stakeholders in the real world.⁶⁾ PCTs combine the scientific rigor of RCTs with the real-life application of observational studies to suggest better solutions that reflect daily clinical practices.⁶⁾ PCTs randomize patients and evaluate the effectiveness rather than efficacy of the intervention (treatment) to investigate the validity of the intervention in a broad routine clinical setting.⁷⁾ PCTs are also referred to as pragmatic randomized clinical trials or embedded pragmatic clinical trials (ePCTs). With the recent increase in the importance and availability of RWE, interest in PCTs has increased worldwide. When searching using PubMed, more than 600 of the included trials conducted between 1977 and 2017 were “pragmatic” in their titles, while more than half of them were concentrated from 2014 to 2017.⁸⁾ In Korea, the National Evidence-based healthcare Collaborating Agency organized a research center to support and fund trials including PCTs in 2019, which has identified the effectiveness of health care technologies and provided evidence for regulatory decisions. Given these circumstances, it is necessary to understand the concept and implementation of PCTs. In this study, we briefly described the main ideas and planning stages of PCTs and the key considerations required to perform them.

CONCEPT OF PCTs

The concept of pragmatism in trials was first presented by Schwartz and Lellouch in 1967.⁹⁾ They classified randomized trials into 2 categories, “explanatory” and “pragmatic,” depending on the conditions of comparison and the level of current practice. Explanatory clinical trials (ECTs) are intended to provide information to understand the efficacy and biological mechanisms of interventions by comparing them in the identical and strict conditions under controlled settings.⁹⁾ By contrast, PCTs aim to assess the effectiveness of interventions in the usual clinical setting and to make relevant decisions.⁷⁾⁹⁾

After the initial concept of PCTs was presented, a variety of factors have been presented by several authors to distinguish ECTs from PCTs (Table 1).^10-16) ECTs determine whether the intervention actually work in controlled environments, while PCTs assess whether the intervention work in real-world settings.¹⁴⁾ Other differences between the 2 representative trials are as follows:

• Participants: ECTs use high levels of inclusion and exclusion criteria to select the appropriate research targets. However, PCTs minimize the exclusion criteria to include a variety of patients visiting the routine clinical sites.

• Settings: ECTs are performed in strictly controlled settings to minimize bias. By contrast, PCTs are performed in multiple institutions that cover the full range of actual clinical trials, not in a single center, to maximize applicability and generalizability.

• Intervention: ECTs provide specific directions, so that the delivery of intervention and compliance with intervention are closely monitored. However, PCTs leave the providers with details of the implementation of intervention as in the normal practice and do not require compliance with the interventions.

• Comparator: ECTs recommend that the therapeutic impact of experimental interventions be compared with placebo, but PCTs generally compare the relative effectiveness of the treatment studied with the treatment currently used in practice instead of the comparing it with a placebo.

• Outcomes: PCTs focus on a broad set of patient-centered outcomes or endpoints; however, ECTs have limited or surrogate measurable outcomes, especially clinical symptoms or biological markers. Consequently, PCTs have high relevance to real life and direct social value than ECTs because they are more likely to be applied to practice at once.¹⁷⁾ Based on these factors, Califf and Sugarman¹¹⁾ defined PCTs as “designed for the primary purpose of informing decision makers regarding the comparative balance of benefits, burdens, and risks of a biomedical or behavioral health intervention at the individual or population level”.

Explanatory and pragmatic trials should not be considered as completely dichotomous, but should be seen as two extremes within the concept of a continuum (Figure 1).¹⁸⁾ To help design the trials that match the researcher's initial intentions by clarifying the concept of explanatory and pragmatic, the PRagmatic Explanatory Continuum Indicator Summary (PRECIS) tool was developed in 2009.¹⁹⁾ In 2015, a revised version, PRECIS-2, was introduced with more loosened requirements.¹⁸⁾ Investigators can identify the nature of the clinical trials they perform by checking for explanatory/pragmatic continuum in nine domains (Table 2).²⁰⁾

STEPS IN CONDUCTING PCTs

The National Institutes of Health (NIH) Collaboration in the US produced the “Living Textbook” to guide the overall considerations when performing PCTs. In particular, the NIH provided essential steps in conducting PCTs using the ePCT Quick Start Guide for convenience. The next 9 steps in the guide represent what the researchers need to consider from planning to initiate the PCTs.

Learning what makes PCTs different

The first step is that researchers understand the characteristics of PCTs. The researchers are required to have abilities to understand the differences between PCTs and traditional ECTs, and to identify the pragmatic components in PCTs by considering the balance between flexibility, adherence, and generalizability. The typical research question in PCTs is whether specific treatments in routine clinical practice are effective in improving the health outcomes in patients compared with other general treatments.²¹⁾ Research questions are likely to be applied to practices immediately¹⁶⁾ because PCTs have direct social value compared with other types of trials, as the questions originate from a variety of stakeholders such as patients, medical personnel, and delivery systems.²¹⁾

Building partnership to ensure a successful trial

In order to conduct a research that supports the objectives of each stakeholder, the participation of key stakeholders in the entire process of PCTs is essential.²²⁾ PCTs generally use EHR data from multiple institutions, which makes it difficult to achieve without partnerships.²³⁾ The main potential stakeholders of patient-centered research such as PCTs include patients, clinicians, health care organization leaders, researchers, policy makers, and regulators.²²⁾ For the partnership before carrying out PCTs, researchers and health care providers should understand and respect the demands, goals, and tasks of each other.²³⁾ Stakeholders participate in PCTs in a number of ways, including advisory groups, collaborations, and co-design.²⁴⁾

Planning for sustainability before launching of PCTs

The investigators need to develop a sustainable plan before the launch of PCTs. Before the designing of trials, researchers have to understand the process and key considerations related to the performance, dissemination, and continuance of the trials. These considerations include the impact of findings, requests of stakeholders, the best trial designs, and the resources needed to deliver the intervention.²⁵⁾ Researchers should develop overall plans for data sharing collected through PCTs and consider the precautions of sharing patients' data to protect their personal information.²⁶⁾

Choosing the right outcomes

The biggest feature of PCTs is that the outcome being selected should be highly relevant to patients, clinical physicians, communities, and real-world clinical decisions. Therefore, the outcome should be discussed with relevant stakeholders in advance.⁹⁾²⁷⁾ The inappropriate outcomes in PCTs, for example, include surrogate outcomes, physiological outcomes, and unimportant outcomes for patients because they have unclear relationships with the endpoints in routine clinical practice and are not suitable to support the actual clinical decisionmaking.²⁸⁾²⁹⁾ After selecting the results and endpoints to be assessed in PCTs, the researchers would investigate routinely collected outcomes from a variety of resources, including EHRs and electronic case report forms while minimizing the interference of the clinical practices.²⁷⁾ However, since information in EHRs may be inaccurate or missing, the validity of the algorithm of the definition of results must be verified and the patient-reported outcomes (PROs) directly provided by the patients must be used.¹²⁾ PROs are important source of information to understand how treatment improves a patients “quality of life and the results that represent a patients” subjective experiences.³⁰⁾ The selection and measurement of outcome should be accurate and reliable without interference in the clinical practices.²⁷⁾³¹⁾

Planning the study design and analysis

Researchers should plan the overall study design such as study settings, randomization, and blinding methods suitable for PCTs and perform a statistical analysis following this step.

• Study sites: The sites in which PCTs are performed should be similar to the actual clinical setting and should be representative.¹²⁾ Sites should be carefully selected because the prevalence of diseases and the availability of medical care may vary greatly depending on the type and region of the institutions.³²⁾ In order to select the sites for PCTs, the heterogeneity, patient characteristics, and disease prevalence of the sites should be assessed.³²⁾ Then, the feasibility and capacity of the study should be confirmed, including interest in study objectives, level of study experience, and number of patients accessible.³²⁾ PCTs should include several heterogeneous sites because if only single or homogeneous sites are included, the test results may not be applicable to patients in various settings.

• Randomization: Randomization can help reduce the selection bias caused by the differences in study groups; any difference in the outcomes can be attributed to the assignment of treatments. Therefore, randomization should be applied in RCTs including PCTs.³³⁾ Even though randomization at both the patient level and cluster (e.g., sites and physicians) level¹²⁾ is possible in PCTs, randomizing at the patient level may affect patients to select intervention, which may result in low recruitment rates.³⁴⁾ Therefore, cluster randomization, which involves the assignment of treatments at the level of physicians, hospitals, or other units handling multiple patients, is useful.³⁴⁾ There are several types of cluster randomization: cluster with crossover randomized design, cluster with partial crossover randomized design, and simple cluster-randomized design.³⁵⁾

• Blinding: Blinding of patients and health care providers during the treatment period is useful to minimize bias and enhance the internal validity in most clinical trials such as ECTs. In PCTs, it is generally not appropriate to use placebo and blinding owing to ethical reasons, preference of non-blinded trials by patients, and the specificity of PCTs.¹⁶⁾ Therefore, the following methods can be recommended to minimize bias when patients or investigators are not blinded in PCTs¹²⁾: 1) use of hard endpoints (i.e., objective endpoints) rather than subjective endpoints, 2) determination of endpoints from blinded health care professionals, and 3) blinding of statisticians and data analysts during the trial and planning of analysis, if possible. If blinding is not possible for the individual patients and investigators, the cluster randomization method can be used.³⁶⁾³⁷⁾ The lack of blinding in PCTs decreases the internal validity, but it will be easier to generalize the test results by increasing the external validity.¹⁶⁾

• Statistical analyses: The statistical analysis methods used mainly in PCTs must be fully understood, and an analysis plan should be made. The analysis method of RCTs can be applied to PCTs, but considerations are required because of the uniqueness of PCTs.³⁸⁾ Intention to treat (ITT) analysis and per-protocol (PP) analysis are required in analyzing the results of clinical trials, and ITT is considered the principal analysis in PCTs.³⁹⁾ Although problems of validity may occur if the randomized patients switch to other alternative treatments during the ITT analysis, ITT is generally recommended as it reduces the risk of selection bias and the unguaranteed covariate balance that can occur in PP analysis.³⁸⁾ However, large amounts of missing data in the ITT analysis can cause a confounding bias.³⁸⁾ To solve this, a follow-up history and the possible reasons for missing data should be recorded for all patients, which is the basis for analyzing the extent and the pattern of missing data. Bias because of missing data can be supplemented using causal inference tools such as a marginal structural model or through inclusion of additional data such as PROs from other sources (e.g., wearable devices).¹²⁾ In addition, PCTs include more heterogeneous populations than RCTs; hence, the responses of subgroups to interventions may differ.³⁸⁾ When evaluating the effectiveness of treatments in subgroup analyses, a priori subgroup analysis with a large sample size should be used to ensure valid estimates of treatment effects and avoid obtaining false positive results.

Considering oversight and monitoring

In this phase, it is necessary to consider whether provision of an informed consent and a data monitoring committee (DMC) are needed to protect the participants of PCTs and to monitor the entire trial. Informed consent is usually required in clinical trials to respect and protect the study participants' privacy. However, an in-depth research consent that is commonly used in traditional RCTs can cause a selection bias and obstruction of real-world clinical practice.⁴⁰⁾⁴¹⁾ Thus, several alternatives were proposed to better integrate consent procedures within a routine clinical setting in PCTs. First, “integrated consent” proposes an approach that integrates clinical and research consent within the same clinical encounter.⁴²⁾ Physicians record the theoretical basis of treatment, the description of the potential risks and benefits of the treatment, and the results in the EHRs and provide normal treatment based on clinical practice. “Targeted consent” involves a process in which the study patients provide minimal information for consent during the consent procedure, owing to which patients can decide whether to register.⁴³⁾ The broadcast approach uses notifications located in prominent locations to inform patients regularly that RCTs may be performed occasionally; therefore, explicit consent is not mandatory.⁴⁴⁾⁴⁵⁾ Finally, a “waiver of consent” involves a process in which the participants are not informed that a study will be conducted and they do not decide whether to participate in the study,⁴¹⁾ which is allowed only in highly limited situations and is generally not performed in PCTs.

In the case of PCTs, the risk-benefit balance and data integrity changes are not only monitored but also special considerations may be applied owing to the practical characteristics of a routine clinical care. DMC is a committee that operates independently of researchers and sponsors in clinical trials.⁴⁶⁾ Investigators should decide whether to use DMC to monitor trials while protecting the study participants.⁴⁷⁾ The US Food and Drug Administration guidelines⁴⁸⁾ recommend the use of DMC if clinical trials have any of the following characteristics: long-term and large-scale multicenter studies, specific safety issues known in advance, and the inclusion of potentially vulnerable populations. The use of DMC in PCTs will ensure scientific validity, as PCTs are mostly multicentered studies comprising a large number of patients from several institutions.⁴⁷⁾

Testing the feasibility of trials

This step is to assess the feasibility of PCTs, which serves to link the actual test design to their performance in the real world. The 2 elements of evaluating feasibility are as follows: evaluating the logistics of clinical trials within the existing health care system and conducting pilot tests on the key aspects of PCTs.⁴⁹⁾ The logistics assessment of PCTs involves documentation of a comprehensive record of PCTs and investigation of the training required for stakeholders.⁵⁰⁾ Pilot testing includes consideration of methods such as access to actual EHR data usage, cooperation with statistical experts to predict problems related to cluster randomization, and consideration of the valid definition of outcomes of interest.⁵¹⁾ The effects of factors related to PCTs, such as informed consent forms, should be assessed.⁵¹⁾ This process confirms the feasibility by exploring potential problems and solutions when PCTs are actually performed.

Preparations for the launching of PCTs

After assessing the feasibility of PCTs, it is necessary to adjust the research design, interventions, and human resources. For example, problems that PRO data are not being routinely collected in the EHR system and problems with the availability of personnel to provide the intervention to patients can be found, which can cause changes in the research flow.⁵²⁾ Hence, a readiness checklist must be implemented to determine whether PCTs can be launched.⁵³⁾

Recruitment plans for the enrollment of participants in PCTs should be devised in this phase. PCTs are recommended to have a well-representative cohort and should include patients representing the real world to enhance the generalizability of the results.¹⁶⁾ For this, researchers should set wide inclusion criteria and minimum exclusion criteria; thus, patients are not excluded from the study even if they have comorbidities or receive concomitant drugs. Unlike other traditional clinical trials, some degree of patient co-enrollment is acceptable in PCTs owing to the improvement in enrollment even though this can lead to unintended interaction and affect the power of the trials.³⁴⁾⁵⁴⁾ In addition, vulnerable populations such as children, pregnant women, and those with disabilities should not be excluded from PCTs because excluding them makes it difficult to identify the actual benefits and risks of the treatments in these patient groups.³⁴⁾ If it is deemed necessary to include the vulnerable populations, it is recommended to include them in PCTs if additional protection is secured.⁵⁵⁾

Preparing for dissemination and implementation

When reporting the findings of PCTs to the stakeholders and public, transparency and applicability are essential in making informed decisions about the practice based on the finding. In 2008, the CONsolidated Standards of Reporting Trials (CONSORT) statement¹⁴⁾ was published and used to assess the validity and reliability of the results and recommended the performance of PCTs. An extension of the CONSORT statement supports extending the following eight CONSORT checklist items for reporting of PCTs: background, participants, intervention, outcomes, sample size, blinding, participant flow, and generalizability of the findings. The dissemination of results in PCTs conceptualizes and measures the implementation outcomes in the real-world setting. An example of dissemination strategies is the use of a framework such as the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, which can emphasize the effectiveness and reach of PCTs.

EXAMPLES OF PCTs

As identified above, PCTs are performed to verify effectiveness in the real-world setting and provide important answers to clinical questions. Since the concept of PCTs was first proposed, some clinical trials labeled pragmatic have been conducted, but most studies that adequately reflect the actual characteristics of pragmatism were conducted just recently. The following five examples are some of the PCTs that have been implemented (Table 3).

First, the Clinical Randomisation of an Antifibrinolytic in Significant Haemorrhage (CRASH-2) study⁵⁶⁾ was a randomized placebo-controlled trial aimed to assess the effectiveness of tranexamic acid in bleeding trauma patients in 274 hospitals from 40 countries. The results of CRASH-2 showed that tranexamic acid treatment within 3 hours of injury significantly reduced the risk of death. Based on the results of CRASH-2, the early administration of tranexamic acid in patients with trauma has been stated in several guidelines.^57-60) The Post-Myocardial Infarction Free Rx and Economic Evaluation trial⁶¹⁾ was a simple cluster-randomized and controlled policy study. This study was developed to assess the effect of providing full prescription drug coverage for statins, β-blockers, angiotensinconverting enzyme inhibitors, and angiotensin II receptor blockers to patients immediately after discharge from the hospital. This is the first randomized study to evaluate the impact of reducing cost-sharing for essential cardiac medications on clinical and economical outcomes. Thrombus aspiration in patients with an ST-elevation myocardial infarction in Scandinavia, Sweden (Thrombus Aspiration in ST-Elevation Myocardial Infarction in Scandinavia; TASTE Trial)⁶²⁾ was performed to test whether thrombus aspiration reduces mortality in patients with an ST-segment elevation myocardial infarction (STEMI) compared with standard care before percutaneous coronary intervention (PCI). However, thrombus aspiration before PCI did not reduce the 30-day mortality in patients with STEMI compared with PCI alone. This trial introduced randomization in the national health registry, thus combining the strengths of a randomized assignment and the benefits of a large-scale registry. Salford Lung Study (SLS) was the first PCT conducted prior to the drug approval.⁶³⁾ SLS assessed the effectiveness and safety of the combination of inhaled corticosteroids (fluticasone furoate) and the long-acting β2-agonist (vilanterol) compared with those of existing chronic obstructive pulmonary disease (COPD) maintenance therapies in patients with COPD in Salford, UK. Results showed that a once a day treatment of a combination of fluticasone furoate and vilanterol lowers the rate of exacerbations in COPD patients without any risk of serious adverse events.⁶⁴⁾ The High-Sensitivity Troponin in the Evaluation of patients with Acute Coronary Syndrome (High-STEACS) trial in Scotland evaluated whether a high-sensitivity cardiac troponin I (hs-cTnI) assay is more effective in reducing myocardial infarction or cardiovascular death at 1 year than a standard troponin assay. This is the first PCT to evaluate the effectiveness of the hs-cTnI assay in patients with a suggested acute coronary syndrome.

The examples of trials mentioned above have various implications to confirm the effectiveness in a real-world setting using a large population from various countries and to expand the data sources that can be used for PCTs. They also reported that the results of PCTs possibly influenced the change in the clinical guidelines, and that PCTs could be used for premarket approval purposes. These trials would be useful examples when conducting PCTs to obtain the RWE of treatments in various countries.

STRENGTH AND LIMITATION OF PCTs

The great advantage of PCTs is that they are designed to test practicability and effectiveness in the actual day-to-day care setting.⁶⁵⁾ Therefore, it is possible to create a basis to assist in the selection of treatment for patients or health care providers¹⁶⁾ and to provide answers to decision makers such as health care policymakers to allocate resources and personnel efficiently.⁷⁾ It is highly likely that physicians and patients will participate in clinical trials owing to the flexibility of patient treatment and the absence of placebo treatment.¹⁶⁾

The main limitation is that PCTs can reduce the internal validity; however, the external validity can be obtained.⁶⁵⁾ Therefore, PCTs should use a large sample size and have a long-term follow-up period; this can create a huge cost burden associated with increasing resources.⁷⁾¹⁶⁾ Because PCTs compare the effectiveness of many interventions in clinical practice, it is possible to understand the overall performance, but it is difficult to determine the exact magnitude of the effect caused by a particular component.⁷⁾ In other words, the fact that the result of PCTs in treatment is effective in routine care does not indicate the exact amount of effectiveness in an actual individual case.⁷⁾ In the case of clinical trials that evaluate new indications or doses of treatment before the drug is launched, they cannot be considered as PCTs because they are required to comply with clinical trial regulations, not with routine care.⁸⁾

CONCLUSIONS

PCTs provide an important basis for decision-making in the real-world clinical practice and policy settings while maintaining the strengths of classical RCTs if they are systematically designed and performed properly. The recent growing interest and importance of RWE have also increased the researchers' attention to the performance of PCTs. To conduct PCTs, it is better to check whether PCTs are feasible and then perform PCTs when they are thought to be appropriate means to obtain RWE while retaining the quality of clinical trials. Although only a few PCTs have been conducted in Korea, the interest in conducting PCTs is increasing lately. In conclusion, to produce RWE related to clinical and policy decisions, academia, physicians, policy makers, and patients must understand the concepts and benefits of PCTs, and they must be encouraged to use PCTs.

ARTICLE INFORMATION

Conflict of Interest

The authors have no financial conflicts of interest.

Author Contributions

Conceptualization: Jeong NY, Choi NK; Investigation: Jeong NY, Kim SH, Lim E, Choi NK; Supervision: Choi NK; Writing - original draft: Jeong NY, Kim SH, Lim E; Writing - review & editing: Jeong NY, Kim SH, Lim E, Choi NK.

Figure 1.

Relationships among the explanatory clinical trials, observational study, and pragmatic clinical trials, including hypothetical PRECIS-2 diagrams. PRECIS-2 diagrams in the figure is referred to Bevan et al.²⁰⁾

PRECIS = PRagmatic Explanatory Continuum Indicator Summary.

REFERENCES

• 1. Silverman SL. From randomized controlled trials to observational studies. Am J Med 2009;122:114–20.ArticlePubMed
• 2. Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials 2009;10:37. ArticlePubMedPMC
• 3. Gill J, Prasad V. Improving observational studies in the era of big data. Lancet 2018;392:716–7.ArticlePubMed
• 4. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000;342:1878–86.ArticlePubMed
• 5. Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet 2002;359:248–52.ArticlePubMed
• 6. Zuidgeest MG, Goetz I, Groenwold RH, Irving E, van Thiel GJ, Grobbee DE; GetReal Work Package 3. Series: pragmatic trials and real world evidence: Paper 1. Introduction. J Clin Epidemiol 2017;88:7–13.ArticlePubMed
• 7. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci 2011;13:217–24.ArticlePubMedPMC
• 8. Dal-Ré R, Janiaud P, Ioannidis JP. Real-world evidence: how pragmatic are randomized controlled trials labeled as pragmatic? BMC Med 2018;16:49. ArticlePubMedPMC
• 9. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis 1967;20:637–48.ArticlePubMed
• 10. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003;290:1624–32.PubMed
• 11. Califf RM, Sugarman J. Exploring the ethical and regulatory issues in pragmatic clinical trials. Clin Trials 2015;12:436–41.ArticlePubMedPMC
• 12. Gamerman V, Cai T; Elsäßer A. Pragmatic randomized clinical trials: best practices and statistical guidance. Health Serv Outcomes Res Methodol 2019;19:23–35.Article
• 13. Zwarenstein M, Treweek S. What kind of randomized trials do we need? J Clin Epidemiol 2009;62:461–3.ArticlePubMed
• 14. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D, CONSORT group; Pragmatic Trials in Healthcare (Practihc) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 2008;337:a2390. ArticlePubMedPMC
• 15. Coats TJ. Future research in emergency medicine: explanation or pragmatism? Large or small? Simple or complex? Emerg Med J 2011;28:1004–7.ArticlePubMed
• 16. MacPherson H. Pragmatic clinical trials. Complement Ther Med 2004;12:136–40.ArticlePubMed
• 17. Borgerson K. Are explanatory trials ethical? Shifting the burden of justification in clinical trial design. Theor Med Bioeth 2013;34:293–308.ArticlePubMed
• 18. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ 2015;350:h2147. ArticlePubMed
• 19. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol 2009;62:464–75.ArticlePubMed
• 20. Bevan A, Biedenbach P, Berger A. Pragmatic randomized trials: considerations for design and implementation [Internet] Bethesda, MD: Evidera; 2019 [cited 2020 May]. Available from https://www.evidera.com/pragmatic-randomized-trials-considerations-for-design-and-implementation/
• 21. Kalkman S, van Thiel G, van der Graaf R, Zuidgeest M, Goetz I, Grobbee D, van Delden J. The social value of pragmatic trials. Bioethics 2017;31:136–43.ArticlePubMed
• 22. Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM, Santa J, Conway PH, Daudelin D, Morrato EH, Leslie LK. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med 2012;27:985–91.ArticlePubMedPMC
• 23. Johnson KE, Tachibana C, Coronado GD, Dember LM, Glasgow RE, Huang SS, Martin PJ, Richards J, Rosenthal G, Septimus E, Simon GE, Solberg L, Suls J, Thompson E, Larson EB. A guide to research partnerships for pragmatic clinical trials. BMJ 2014;349:g6826. ArticlePubMedPMC
• 24. Fraser J, Moloney R, Tambor E, Tuzzio L. Building partnerships to ensure a successful trial: deciding who to engage. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2020 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/engaging-stakeholders/deciding-who-to-engagev2/
• 25. Chambers D, Coronado G, Green B, Jarvik J, Septimus E, Tuzzio L, Zatzick D. Designing with implementation and dissemination in mind: key considerations. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2018 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/design/designing-implementation-dissemination-mind-top/designing-with-implementation-and-dissemination-in-mind-2/
• 26. Simon G, Ramsberg J, Coronado G, Septimus E, DeBar L, Vazquez M, Dember LM, Vollmer WM, Green B, Zatzick D, Huang SS, Hernandez A, Jarvik J, Platt R, Mor V. Data sharing and embedded research: introduction. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2019 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/dissemination/data-share-top/data-sharing-andembedded-research-introduction/
• 27. Welsing PM, Oude Rengerink K, Collier S, Eckert L, van Smeden M, Ciaglia A, Nachbaur G, Trelle S, Taylor AJ, Egger M, Goetz I; Work Package 3 of the GetReal Consortium. Series: pragmatic trials and real world evidence: Paper 6. Outcome measures in the real world. J Clin Epidemiol 2017;90:99–107.ArticlePubMed
• 28. Curtis L, Hernandez A, Weinfurt K. Choosing and specifying endpoints and outcomes: meaningful endpoints. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2020 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/design/choosing-specifying-end-points-outcomes/meaningful-endpoints/
• 29. Murad MH, Shah ND, Van Houten HK, Ziegenfuss JY, Deming JR, Beebe TJ, Smith SA, Guyatt GH, Montori VM. Individuals with diabetes preferred that future trials use patient-important outcomes and provide pragmatic inferences. J Clin Epidemiol 2011;64:743–8.ArticlePubMed
• 30. Basch E. New frontiers in patient-reported outcomes: adverse event reporting, comparative effectiveness, and quality assessment. Annu Rev Med 2014;65:307–17.ArticlePubMed
• 31. Walton MK, Powers JH 3rd, Hobart J, Patrick D, Marquis P, Vamvakas S, Isaac M, Molsen E, Cano S, Burke LB; International Society for Pharmacoeconomics and Outcomes Research Task Force for Clinical Outcomes Assessment. Clinical outcome assessments: conceptual foundation-report of the ISPOR Clinical Outcomes Assessment - Emerging Good Practices for Outcomes Research Task Force. Value Health 2015;18:741–52.ArticlePubMedPMC
• 32. Worsley SD, Oude Rengerink K, Irving E, Lejeune S, Mol K, Collier S, Groenwold RH, Enters-Weijnen C, Egger M, Rhodes T; GetReal Work Package 3. Series: pragmatic trials and real world evidence: Paper 2. Setting, sites, and investigator selection. J Clin Epidemiol 2017;88:14–20.ArticlePubMed
• 33. Bulpitt CJ. Randomised Controlled Clinical Trials. New York, NY, Springer Science & Business Media; 2012.
• 34. Oude Rengerink K, Kalkman S, Collier S, Ciaglia A, Worsley SD, Lightbourne A, Eckert L, Groenwold RH, Grobbee DE, Irving EA; Work Package 3 of the GetReal consortium. Series: pragmatic trials and real world evidence: Paper 3. Patient selection challenges and consequences. J Clin Epidemiol 2017;89:173–80.ArticlePubMed
• 35. Cook AJ, Delong E, Murray DM, Vollmer WM, Heagerty PJ. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core. Clin Trials 2016;13:504–12.ArticlePubMedPMC
• 36. Godwin M, Ruhland L, Casson I, MacDonald S, Delva D, Birtwhistle R, Lam M, Seguin R. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol 2003;3:28. ArticlePubMedPMC
• 37. Fransen GA, van Marrewijk CJ, Mujakovic S, Muris JW, Laheij RJ, Numans ME, de Wit NJ, Samsom M, Jansen JB, Knottnerus JA. Pragmatic trials in primary care. Methodological challenges and solutions demonstrated by the DIAMOND-study. BMC Med Res Methodol 2007;7:16. ArticlePubMedPMC
• 38. Jiang D, Chen K, Mukhopadhyay S, Katta N, Zhang L. Review of Statistical Issues in Pragmatic Clinical Trials in Current Drug Development Environment. In: Contemporary Biostatistics with Biopharmaceutical Applications. Cham, Switzerland, Springer; 2019. p. 285–98.
• 39. Gupta SK. Intention-to-treat concept: a review. Perspect Clin Res 2011;2:109–12.ArticlePubMedPMC
• 40. Sugarman J, Califf RM. Ethics and regulatory complexities for pragmatic clinical trials. JAMA 2014;311:2381–2.ArticlePubMed
• 41. McKinney RE Jr, Beskow LM, Ford DE, Lantos JD, McCall J, Patrick-Lake B, Pletcher MJ, Rath B, Schmidt H, Weinfurt K. Use of altered informed consent in pragmatic clinical research. Clin Trials 2015;12:494–502.ArticlePubMedPMC
• 42. Kim SY, Miller FG. Informed consent for pragmatic trials--the integrated consent model. N Engl J Med 2014;370:769–72.ArticlePubMed
• 43. Wendler D. “Targeted” consent for pragmatic clinical trials. J Gen Intern Med 2015;30:679–82.ArticlePubMedPMC
• 44. Faden R, Kass N, Whicher D, Stewart W, Tunis S. Ethics and informed consent for comparative effectiveness research with prospective electronic clinical data. Med Care 2013;51:S53–7.ArticlePubMed
• 45. Faden RR, Beauchamp TL, Kass NE. Informed consent, comparative effectiveness, and learning health care. N Engl J Med 2014;370:766–8.ArticlePubMed
• 46. D'Agostino RB; Sullivan LM, Massaro J. Wiley Encyclopedia of Clinical Trials. Hoboken, NJ, John Wiley & Sons, Inc; 2007.
• 47. Ellenberg S, Sugarman J, Zatzick D. Data and safety monitoring: which PCTs should have a DMC? In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2020 [cited 2020 May 30]. Available from: https://rethinkingclinicaltrials.org/chapters/design/planning-data-safety-monitoring/which-pcts-should-have-a-dmc/
• 48. Food and Drug Administration. FDA Guidance for Clinical Trial Sponsors On the Establishment and Operation of Clinical Trial Data Monitoring Committees. Rockville, MD, Food and Drug Administration; 2006.
• 49. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Assessing feasibility: introduction. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2018 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/assessing-feasibility/assessing-feasibility-introduction/
• 50. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Assessing feasibility: developing the trial documentation. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2018 [cited 2020 May 30]. Available from https:// rethinkingclinicaltrials.org/chapters/conduct/assessing-feasibility/developing-the-trial-documentation/
• 51. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Assessing feasibility: pilot testing. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2020 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/assessing-feasibility/pilot-testing/
• 52. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Study startup: introduction. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2020 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/startup/startup-introduction/
• 53. DeBar L, Jarvik J, Tuzzio L, Vazquez M. Study Startup: Implementation Readiness Checklist. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet] Bethesda, MD: NIH Health Care Systems Research Collaboratory; 2020 [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/startup/startup-implementation/
• 54. Cook DJ, Ferguson ND, Hand L, Austin P, Zhou Q, Adhikari NK, Danesh V, Arabi Y, Matte AL, Clarke FE, Mehta S, Smith O, Wise MP, Friedrich JO, Keenan SP, Hanna S, Meade MO; OSCILLation for ARDS Treated Early Investigators; Canadian Critical Care Trials Group. Coenrollment in a randomized trial of highfrequency oscillation: prevalence, patterns, predictors, and outcomes. Crit Care Med 2015;43:328–38.PubMed
• 55. Welch MJ, Lally R, Miller JE, Pittman S, Brodsky L, Caplan AL, Uhlenbrauck G, Louzao DM, Fischer JH, Wilfond B. The ethics and regulatory landscape of including vulnerable populations in pragmatic clinical trials. Clin Trials 2015;12:503–10.ArticlePubMedPMC
• 56. CRASH-2 trial collaborators; Shakur H, Roberts I, Bautista R, Caballero J, Coats T, Dewan Y, El-Sayed H, Gogichaishvili T, Gupta S, Herrera J, Hunt B, Iribhogbe P, Izurieta M, Khamis H, Komolafe E, Marrero MA, Mejía-Mantilla J, Miranda J, Morales C, Olaomi O, Olldashi F, Perel P, Peto R, Ramana PV, Ravi RR, Yutthakasemsunt S. Effects of tranexamic acid on death, vascular occlusive events, and blood transfusion in trauma patients with significant haemorrhage (CRASH-2): a randomised, placebo-controlled trial. Lancet 2010;376:23–32.ArticlePubMed
• 57. Wada H, Thachil J, Di Nisio M, Mathew P, Kurosawa S, Gando S, Kim HK, Nielsen JD, Dempfle CE, Levi M, Toh CH; The Scientific Standardization Committee on DIC of the International Society on Thrombosis Haemostasis. Guidance for diagnosis and treatment of DIC from harmonization of the recommendations from three guidelines. J Thromb Haemost 2013;11:761–7.Article
• 58. Hunt BJ, Allard S, Keeling D, Norfolk D, Stanworth SJ, Pendry K; British Committee for Standards in Haematology. A practical guideline for the haematological management of major haemorrhage. Br J Haematol 2015;170:788–803.ArticlePubMed
• 59. Rossaint R, Bouillon B, Cerny V, Coats TJ, Duranteau J, Fernández-Mondéjar E, Filipescu D, Hunt BJ, Komadina R, Nardi G, Neugebauer EA, Ozier Y, Riddez L, Schultz A, Vincent JL, Spahn DR. The European guideline on management of major bleeding and coagulopathy following trauma: fourth edition. Crit Care 2016;20:100. ArticlePubMedPMC
• 60. National Clinical Guideline Centre. Major Trauma: Assessment and Initial Management. London, National Institute for Health and Care Excellence; 2016.
• 61. Choudhry NK, Avorn J, Glynn RJ, Antman EM, Schneeweiss S, Toscano M, Reisman L, Fernandes J, Spettell C, Lee JL, Levin R, Brennan T, Shrank WH; Post-Myocardial Infarction Free Rx Event and Economic Evaluation (MI FREEE) Trial. Full coverage for preventive medications after myocardial infarction. N Engl J Med 2011;365:2088–97.ArticlePubMed
• 62. Fröbert O, Lagerqvist B, Olivecrona GK, Omerovic E, Gudnason T, Maeng M, Aasa M, Angerås O, Calais F, Danielewicz M, Erlinge D, Hellsten L, Jensen U, Johansson AC, Kåregren A, Nilsson J, Robertson L, Sandhall L, Sjögren I, Ostlund O, Harnek J, James SK, TASTE Trial. Thrombus aspiration during STsegment elevation myocardial infarction. N Engl J Med 2013;369:1587–97.ArticlePubMed
• 63. Bakerly ND, Woodcock A, New JP, Gibson JM, Wu W, Leather D, Vestbo J. The Salford Lung Study protocol: a pragmatic, randomised phase III real-world effectiveness trial in chronic obstructive pulmonary disease. Respir Res 2015;16:101. ArticlePubMedPMC
• 64. Vestbo J, Leather D, Diar Bakerly N, New J, Gibson JM, McCorkindale S, Collier S, Crawford J, Frith L, Harvey C, Svedsater H, Woodcock A; Salford Lung Study Investigators. Effectiveness of fluticasone furoate-vilanterol for COPD in clinical practice. N Engl J Med 2016;375:1253–60.ArticlePubMed
• 65. Kumar A, Rana V. Pragmatic trials: importance and impact. J Endocrinol Thyroid Res 2017;2:555594.
• 66. Shah AS, Anand A, Strachan FE, Ferry AV, Lee KK, Chapman AR, Sandeman D, Stables CL, Adamson PD, Andrews JP, Anwar MS, Hung J, Moss AJ, O'Brien R, Berry C, Findlay I, Walker S, Cruickshank A, Reid A, Gray A, Collinson PO, Apple FS, McAllister DA, Maguire D, Fox KA, Newby DE, Tuck C, Harkess R, Parker RA, Keerie C, Weir CJ; Mills NLHigh-STEACS Investigators. High-sensitivity troponin in the evaluation of patients with suspected acute coronary syndrome: a stepped-wedge, cluster-randomised controlled trial. Lancet 2018;392:919–28.ArticlePubMedPMC

Figure & Data

References

Citations

Citations to this article as recorded by  

• A scoping review of the Choice and Partnership Approach in child and adolescent mental health services
  Kathleen Pajer, Carlos Pastrana, William Gardner, Aditi Sivakumar, Ann York
  Journal of Child Health Care.2023; 27(4): 707.     CrossRef