Pragmatic Clinical Trials for Real-World Evidence: Concept and Implementation
Article information
Abstract
The importance of real-world evidence (RWE) has been highlighted in recent years, and the limitations of the classical randomized controlled trials, also known as explanatory clinical trials (ECTs), have been emphasized. Post-marketing observational studies have several problems, such as biases and incomparability between patient groups, and RWE can only be obtained after a certain period. Therefore, pragmatic clinical trials (PCTs) have garnered attention as an alternative to obtaining scientifically robust RWE in a relatively short time. PCTs are clinical trials that have a pragmatic concept, i.e., the opposite of ECTs and are intended to help decision makers by evaluating the effectiveness of interventions in routine clinical practice. The characteristics of PCTs are the inclusion of various patients in clinical practice, recruitment of patients in heterogeneous settings, and comparison with actual clinical treatments rather than a placebo. Thus, the results of PCTs are likely to be generalized and can have a direct impact on clinical and policy decision-making. This study aimed to describe the characteristics and definitions of PCTs compared with those of ECTs and to highlight the important considerations in the planning process of PCTs. To perform PCTs for the purpose of obtaining RWE, the contents covered in this study will be helpful.
INTRODUCTION
Traditionally, randomized controlled trials (RCTs) in controlled settings have been considered the gold standard for studying the efficacy of health interventions.1) It is the backbone of clinical research because randomization in well-defined and controlled settings can improve the internal validity and reduce bias, but these settings tend to overestimate the benefits and underestimate the risks.2) Recently, there has been great interest in conducting post-launch observational studies that increase the external validity using real-world data (e.g., claims data and electronic health records [EHRs]) including a large population.1)3) However, several challenges have been reported when conducting these studies, including bias (e.g., selection bias), inaccuracy of diagnostic codes, and incomparability between patient groups.4)5) In terms of real-world evidence (RWE) delivery, pragmatic clinical trials (PCTs) are proposed as a method to provide evidence in a relatively shorter time to solve problems faced by multiple stakeholders in the real world.6) PCTs combine the scientific rigor of RCTs with the real-life application of observational studies to suggest better solutions that reflect daily clinical practices.6) PCTs randomize patients and evaluate the effectiveness rather than efficacy of the intervention (treatment) to investigate the validity of the intervention in a broad routine clinical setting.7) PCTs are also referred to as pragmatic randomized clinical trials or embedded pragmatic clinical trials (ePCTs). With the recent increase in the importance and availability of RWE, interest in PCTs has increased worldwide. When searching using PubMed, more than 600 of the included trials conducted between 1977 and 2017 were “pragmatic” in their titles, while more than half of them were concentrated from 2014 to 2017.8) In Korea, the National Evidence-based healthcare Collaborating Agency organized a research center to support and fund trials including PCTs in 2019, which has identified the effectiveness of health care technologies and provided evidence for regulatory decisions. Given these circumstances, it is necessary to understand the concept and implementation of PCTs. In this study, we briefly described the main ideas and planning stages of PCTs and the key considerations required to perform them.
CONCEPT OF PCTs
The concept of pragmatism in trials was first presented by Schwartz and Lellouch in 1967.9) They classified randomized trials into 2 categories, “explanatory” and “pragmatic,” depending on the conditions of comparison and the level of current practice. Explanatory clinical trials (ECTs) are intended to provide information to understand the efficacy and biological mechanisms of interventions by comparing them in the identical and strict conditions under controlled settings.9) By contrast, PCTs aim to assess the effectiveness of interventions in the usual clinical setting and to make relevant decisions.7)9)
After the initial concept of PCTs was presented, a variety of factors have been presented by several authors to distinguish ECTs from PCTs (Table 1).10-16) ECTs determine whether the intervention actually work in controlled environments, while PCTs assess whether the intervention work in real-world settings.14) Other differences between the 2 representative trials are as follows:
Participants: ECTs use high levels of inclusion and exclusion criteria to select the appropriate research targets. However, PCTs minimize the exclusion criteria to include a variety of patients visiting the routine clinical sites.
Settings: ECTs are performed in strictly controlled settings to minimize bias. By contrast, PCTs are performed in multiple institutions that cover the full range of actual clinical trials, not in a single center, to maximize applicability and generalizability.
Intervention: ECTs provide specific directions, so that the delivery of intervention and compliance with intervention are closely monitored. However, PCTs leave the providers with details of the implementation of intervention as in the normal practice and do not require compliance with the interventions.
Comparator: ECTs recommend that the therapeutic impact of experimental interventions be compared with placebo, but PCTs generally compare the relative effectiveness of the treatment studied with the treatment currently used in practice instead of the comparing it with a placebo.
Outcomes: PCTs focus on a broad set of patient-centered outcomes or endpoints; however, ECTs have limited or surrogate measurable outcomes, especially clinical symptoms or biological markers. Consequently, PCTs have high relevance to real life and direct social value than ECTs because they are more likely to be applied to practice at once.17) Based on these factors, Califf and Sugarman11) defined PCTs as “designed for the primary purpose of informing decision makers regarding the comparative balance of benefits, burdens, and risks of a biomedical or behavioral health intervention at the individual or population level”.
Explanatory and pragmatic trials should not be considered as completely dichotomous, but should be seen as two extremes within the concept of a continuum (Figure 1).18) To help design the trials that match the researcher's initial intentions by clarifying the concept of explanatory and pragmatic, the PRagmatic Explanatory Continuum Indicator Summary (PRECIS) tool was developed in 2009.19) In 2015, a revised version, PRECIS-2, was introduced with more loosened requirements.18) Investigators can identify the nature of the clinical trials they perform by checking for explanatory/pragmatic continuum in nine domains (Table 2).20)
STEPS IN CONDUCTING PCTs
The National Institutes of Health (NIH) Collaboration in the US produced the “Living Textbook” to guide the overall considerations when performing PCTs. In particular, the NIH provided essential steps in conducting PCTs using the ePCT Quick Start Guide for convenience. The next 9 steps in the guide represent what the researchers need to consider from planning to initiate the PCTs.
Learning what makes PCTs different
The first step is that researchers understand the characteristics of PCTs. The researchers are required to have abilities to understand the differences between PCTs and traditional ECTs, and to identify the pragmatic components in PCTs by considering the balance between flexibility, adherence, and generalizability. The typical research question in PCTs is whether specific treatments in routine clinical practice are effective in improving the health outcomes in patients compared with other general treatments.21) Research questions are likely to be applied to practices immediately16) because PCTs have direct social value compared with other types of trials, as the questions originate from a variety of stakeholders such as patients, medical personnel, and delivery systems.21)
Building partnership to ensure a successful trial
In order to conduct a research that supports the objectives of each stakeholder, the participation of key stakeholders in the entire process of PCTs is essential.22) PCTs generally use EHR data from multiple institutions, which makes it difficult to achieve without partnerships.23) The main potential stakeholders of patient-centered research such as PCTs include patients, clinicians, health care organization leaders, researchers, policy makers, and regulators.22) For the partnership before carrying out PCTs, researchers and health care providers should understand and respect the demands, goals, and tasks of each other.23) Stakeholders participate in PCTs in a number of ways, including advisory groups, collaborations, and co-design.24)
Planning for sustainability before launching of PCTs
The investigators need to develop a sustainable plan before the launch of PCTs. Before the designing of trials, researchers have to understand the process and key considerations related to the performance, dissemination, and continuance of the trials. These considerations include the impact of findings, requests of stakeholders, the best trial designs, and the resources needed to deliver the intervention.25) Researchers should develop overall plans for data sharing collected through PCTs and consider the precautions of sharing patients' data to protect their personal information.26)
Choosing the right outcomes
The biggest feature of PCTs is that the outcome being selected should be highly relevant to patients, clinical physicians, communities, and real-world clinical decisions. Therefore, the outcome should be discussed with relevant stakeholders in advance.9)27) The inappropriate outcomes in PCTs, for example, include surrogate outcomes, physiological outcomes, and unimportant outcomes for patients because they have unclear relationships with the endpoints in routine clinical practice and are not suitable to support the actual clinical decisionmaking.28)29) After selecting the results and endpoints to be assessed in PCTs, the researchers would investigate routinely collected outcomes from a variety of resources, including EHRs and electronic case report forms while minimizing the interference of the clinical practices.27) However, since information in EHRs may be inaccurate or missing, the validity of the algorithm of the definition of results must be verified and the patient-reported outcomes (PROs) directly provided by the patients must be used.12) PROs are important source of information to understand how treatment improves a patients “quality of life and the results that represent a patients” subjective experiences.30) The selection and measurement of outcome should be accurate and reliable without interference in the clinical practices.27)31)
Planning the study design and analysis
Researchers should plan the overall study design such as study settings, randomization, and blinding methods suitable for PCTs and perform a statistical analysis following this step.
Study sites: The sites in which PCTs are performed should be similar to the actual clinical setting and should be representative.12) Sites should be carefully selected because the prevalence of diseases and the availability of medical care may vary greatly depending on the type and region of the institutions.32) In order to select the sites for PCTs, the heterogeneity, patient characteristics, and disease prevalence of the sites should be assessed.32) Then, the feasibility and capacity of the study should be confirmed, including interest in study objectives, level of study experience, and number of patients accessible.32) PCTs should include several heterogeneous sites because if only single or homogeneous sites are included, the test results may not be applicable to patients in various settings.
Randomization: Randomization can help reduce the selection bias caused by the differences in study groups; any difference in the outcomes can be attributed to the assignment of treatments. Therefore, randomization should be applied in RCTs including PCTs.33) Even though randomization at both the patient level and cluster (e.g., sites and physicians) level12) is possible in PCTs, randomizing at the patient level may affect patients to select intervention, which may result in low recruitment rates.34) Therefore, cluster randomization, which involves the assignment of treatments at the level of physicians, hospitals, or other units handling multiple patients, is useful.34) There are several types of cluster randomization: cluster with crossover randomized design, cluster with partial crossover randomized design, and simple cluster-randomized design.35)
Blinding: Blinding of patients and health care providers during the treatment period is useful to minimize bias and enhance the internal validity in most clinical trials such as ECTs. In PCTs, it is generally not appropriate to use placebo and blinding owing to ethical reasons, preference of non-blinded trials by patients, and the specificity of PCTs.16) Therefore, the following methods can be recommended to minimize bias when patients or investigators are not blinded in PCTs12): 1) use of hard endpoints (i.e., objective endpoints) rather than subjective endpoints, 2) determination of endpoints from blinded health care professionals, and 3) blinding of statisticians and data analysts during the trial and planning of analysis, if possible. If blinding is not possible for the individual patients and investigators, the cluster randomization method can be used.36)37) The lack of blinding in PCTs decreases the internal validity, but it will be easier to generalize the test results by increasing the external validity.16)
Statistical analyses: The statistical analysis methods used mainly in PCTs must be fully understood, and an analysis plan should be made. The analysis method of RCTs can be applied to PCTs, but considerations are required because of the uniqueness of PCTs.38) Intention to treat (ITT) analysis and per-protocol (PP) analysis are required in analyzing the results of clinical trials, and ITT is considered the principal analysis in PCTs.39) Although problems of validity may occur if the randomized patients switch to other alternative treatments during the ITT analysis, ITT is generally recommended as it reduces the risk of selection bias and the unguaranteed covariate balance that can occur in PP analysis.38) However, large amounts of missing data in the ITT analysis can cause a confounding bias.38) To solve this, a follow-up history and the possible reasons for missing data should be recorded for all patients, which is the basis for analyzing the extent and the pattern of missing data. Bias because of missing data can be supplemented using causal inference tools such as a marginal structural model or through inclusion of additional data such as PROs from other sources (e.g., wearable devices).12) In addition, PCTs include more heterogeneous populations than RCTs; hence, the responses of subgroups to interventions may differ.38) When evaluating the effectiveness of treatments in subgroup analyses, a priori subgroup analysis with a large sample size should be used to ensure valid estimates of treatment effects and avoid obtaining false positive results.
Considering oversight and monitoring
In this phase, it is necessary to consider whether provision of an informed consent and a data monitoring committee (DMC) are needed to protect the participants of PCTs and to monitor the entire trial. Informed consent is usually required in clinical trials to respect and protect the study participants' privacy. However, an in-depth research consent that is commonly used in traditional RCTs can cause a selection bias and obstruction of real-world clinical practice.40)41) Thus, several alternatives were proposed to better integrate consent procedures within a routine clinical setting in PCTs. First, “integrated consent” proposes an approach that integrates clinical and research consent within the same clinical encounter.42) Physicians record the theoretical basis of treatment, the description of the potential risks and benefits of the treatment, and the results in the EHRs and provide normal treatment based on clinical practice. “Targeted consent” involves a process in which the study patients provide minimal information for consent during the consent procedure, owing to which patients can decide whether to register.43) The broadcast approach uses notifications located in prominent locations to inform patients regularly that RCTs may be performed occasionally; therefore, explicit consent is not mandatory.44)45) Finally, a “waiver of consent” involves a process in which the participants are not informed that a study will be conducted and they do not decide whether to participate in the study,41) which is allowed only in highly limited situations and is generally not performed in PCTs.
In the case of PCTs, the risk-benefit balance and data integrity changes are not only monitored but also special considerations may be applied owing to the practical characteristics of a routine clinical care. DMC is a committee that operates independently of researchers and sponsors in clinical trials.46) Investigators should decide whether to use DMC to monitor trials while protecting the study participants.47) The US Food and Drug Administration guidelines48) recommend the use of DMC if clinical trials have any of the following characteristics: long-term and large-scale multicenter studies, specific safety issues known in advance, and the inclusion of potentially vulnerable populations. The use of DMC in PCTs will ensure scientific validity, as PCTs are mostly multicentered studies comprising a large number of patients from several institutions.47)
Testing the feasibility of trials
This step is to assess the feasibility of PCTs, which serves to link the actual test design to their performance in the real world. The 2 elements of evaluating feasibility are as follows: evaluating the logistics of clinical trials within the existing health care system and conducting pilot tests on the key aspects of PCTs.49) The logistics assessment of PCTs involves documentation of a comprehensive record of PCTs and investigation of the training required for stakeholders.50) Pilot testing includes consideration of methods such as access to actual EHR data usage, cooperation with statistical experts to predict problems related to cluster randomization, and consideration of the valid definition of outcomes of interest.51) The effects of factors related to PCTs, such as informed consent forms, should be assessed.51) This process confirms the feasibility by exploring potential problems and solutions when PCTs are actually performed.
Preparations for the launching of PCTs
After assessing the feasibility of PCTs, it is necessary to adjust the research design, interventions, and human resources. For example, problems that PRO data are not being routinely collected in the EHR system and problems with the availability of personnel to provide the intervention to patients can be found, which can cause changes in the research flow.52) Hence, a readiness checklist must be implemented to determine whether PCTs can be launched.53)
Recruitment plans for the enrollment of participants in PCTs should be devised in this phase. PCTs are recommended to have a well-representative cohort and should include patients representing the real world to enhance the generalizability of the results.16) For this, researchers should set wide inclusion criteria and minimum exclusion criteria; thus, patients are not excluded from the study even if they have comorbidities or receive concomitant drugs. Unlike other traditional clinical trials, some degree of patient co-enrollment is acceptable in PCTs owing to the improvement in enrollment even though this can lead to unintended interaction and affect the power of the trials.34)54) In addition, vulnerable populations such as children, pregnant women, and those with disabilities should not be excluded from PCTs because excluding them makes it difficult to identify the actual benefits and risks of the treatments in these patient groups.34) If it is deemed necessary to include the vulnerable populations, it is recommended to include them in PCTs if additional protection is secured.55)
Preparing for dissemination and implementation
When reporting the findings of PCTs to the stakeholders and public, transparency and applicability are essential in making informed decisions about the practice based on the finding. In 2008, the CONsolidated Standards of Reporting Trials (CONSORT) statement14) was published and used to assess the validity and reliability of the results and recommended the performance of PCTs. An extension of the CONSORT statement supports extending the following eight CONSORT checklist items for reporting of PCTs: background, participants, intervention, outcomes, sample size, blinding, participant flow, and generalizability of the findings. The dissemination of results in PCTs conceptualizes and measures the implementation outcomes in the real-world setting. An example of dissemination strategies is the use of a framework such as the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, which can emphasize the effectiveness and reach of PCTs.
EXAMPLES OF PCTs
As identified above, PCTs are performed to verify effectiveness in the real-world setting and provide important answers to clinical questions. Since the concept of PCTs was first proposed, some clinical trials labeled pragmatic have been conducted, but most studies that adequately reflect the actual characteristics of pragmatism were conducted just recently. The following five examples are some of the PCTs that have been implemented (Table 3).
First, the Clinical Randomisation of an Antifibrinolytic in Significant Haemorrhage (CRASH-2) study56) was a randomized placebo-controlled trial aimed to assess the effectiveness of tranexamic acid in bleeding trauma patients in 274 hospitals from 40 countries. The results of CRASH-2 showed that tranexamic acid treatment within 3 hours of injury significantly reduced the risk of death. Based on the results of CRASH-2, the early administration of tranexamic acid in patients with trauma has been stated in several guidelines.57-60) The Post-Myocardial Infarction Free Rx and Economic Evaluation trial61) was a simple cluster-randomized and controlled policy study. This study was developed to assess the effect of providing full prescription drug coverage for statins, β-blockers, angiotensinconverting enzyme inhibitors, and angiotensin II receptor blockers to patients immediately after discharge from the hospital. This is the first randomized study to evaluate the impact of reducing cost-sharing for essential cardiac medications on clinical and economical outcomes. Thrombus aspiration in patients with an ST-elevation myocardial infarction in Scandinavia, Sweden (Thrombus Aspiration in ST-Elevation Myocardial Infarction in Scandinavia; TASTE Trial)62) was performed to test whether thrombus aspiration reduces mortality in patients with an ST-segment elevation myocardial infarction (STEMI) compared with standard care before percutaneous coronary intervention (PCI). However, thrombus aspiration before PCI did not reduce the 30-day mortality in patients with STEMI compared with PCI alone. This trial introduced randomization in the national health registry, thus combining the strengths of a randomized assignment and the benefits of a large-scale registry. Salford Lung Study (SLS) was the first PCT conducted prior to the drug approval.63) SLS assessed the effectiveness and safety of the combination of inhaled corticosteroids (fluticasone furoate) and the long-acting β2-agonist (vilanterol) compared with those of existing chronic obstructive pulmonary disease (COPD) maintenance therapies in patients with COPD in Salford, UK. Results showed that a once a day treatment of a combination of fluticasone furoate and vilanterol lowers the rate of exacerbations in COPD patients without any risk of serious adverse events.64) The High-Sensitivity Troponin in the Evaluation of patients with Acute Coronary Syndrome (High-STEACS) trial in Scotland evaluated whether a high-sensitivity cardiac troponin I (hs-cTnI) assay is more effective in reducing myocardial infarction or cardiovascular death at 1 year than a standard troponin assay. This is the first PCT to evaluate the effectiveness of the hs-cTnI assay in patients with a suggested acute coronary syndrome.
The examples of trials mentioned above have various implications to confirm the effectiveness in a real-world setting using a large population from various countries and to expand the data sources that can be used for PCTs. They also reported that the results of PCTs possibly influenced the change in the clinical guidelines, and that PCTs could be used for premarket approval purposes. These trials would be useful examples when conducting PCTs to obtain the RWE of treatments in various countries.
STRENGTH AND LIMITATION OF PCTs
The great advantage of PCTs is that they are designed to test practicability and effectiveness in the actual day-to-day care setting.65) Therefore, it is possible to create a basis to assist in the selection of treatment for patients or health care providers16) and to provide answers to decision makers such as health care policymakers to allocate resources and personnel efficiently.7) It is highly likely that physicians and patients will participate in clinical trials owing to the flexibility of patient treatment and the absence of placebo treatment.16)
The main limitation is that PCTs can reduce the internal validity; however, the external validity can be obtained.65) Therefore, PCTs should use a large sample size and have a long-term follow-up period; this can create a huge cost burden associated with increasing resources.7)16) Because PCTs compare the effectiveness of many interventions in clinical practice, it is possible to understand the overall performance, but it is difficult to determine the exact magnitude of the effect caused by a particular component.7) In other words, the fact that the result of PCTs in treatment is effective in routine care does not indicate the exact amount of effectiveness in an actual individual case.7) In the case of clinical trials that evaluate new indications or doses of treatment before the drug is launched, they cannot be considered as PCTs because they are required to comply with clinical trial regulations, not with routine care.8)
CONCLUSIONS
PCTs provide an important basis for decision-making in the real-world clinical practice and policy settings while maintaining the strengths of classical RCTs if they are systematically designed and performed properly. The recent growing interest and importance of RWE have also increased the researchers' attention to the performance of PCTs. To conduct PCTs, it is better to check whether PCTs are feasible and then perform PCTs when they are thought to be appropriate means to obtain RWE while retaining the quality of clinical trials. Although only a few PCTs have been conducted in Korea, the interest in conducting PCTs is increasing lately. In conclusion, to produce RWE related to clinical and policy decisions, academia, physicians, policy makers, and patients must understand the concepts and benefits of PCTs, and they must be encouraged to use PCTs.
Notes
Conflict of Interest
The authors have no financial conflicts of interest.
Author Contributions
Conceptualization: Jeong NY, Choi NK; Investigation: Jeong NY, Kim SH, Lim E, Choi NK; Supervision: Choi NK; Writing - original draft: Jeong NY, Kim SH, Lim E; Writing - review & editing: Jeong NY, Kim SH, Lim E, Choi NK.