Randomized Controlled Trials: A Primer for Neuro-Ophthalmologists

Editorial Randomized Controlled Trials: A Primer for Neuro-Ophthalmologists Heather E. Moss, MD, PhD, Jing Cao, PhD, Stacy L. Pineles, MD R andomized controlled trials (RCT) are critically important in development of treatments, gaining regulatory approval for treatments, and payment for treatments. A single RCT is classified as level 2 evidence and a systematic review of more than one RCT is classified as Level 1 evidence by the Oxford Centre for Evidence Based Medicine (1–3). Given the importance of this study design to advancing treatment in medicine, and the expanding opportunities for participation in multi-center RCTs through the Neuro-Ophthalmology Research Disease Investigator Consortium, it is important that neuro-ophthalmologists be familiar with the key considerations in RCT design, administration, and interpretation. This article, presented as a companion to “Randomized Controlled Phase 2a Study of RPh201 in Previous Nonarteritic Anterior Ischemic Optic Neuropathy” by Rath et al in a previous issue of JNO (4), presents an overview of RCTs aimed at the practicing neuro-ophthalmologist. It is important to distinguish the clinical research trial phase, which indicates the stage of testing during drug development, from study design, with RCT being the typical study design for Phase 2 and 3 clinical trials, but not Stage 1 (safety, dosing) or Stage 4 (surveillance). RPh201, the compound in Rath et al’s study, was previously investigated in a Phase 1 study, which tested multiple doses in healthy human volunteers with a primary goal of assessing safety. Rath et al’s study is Phase 2a (clinicaltrials.gov NCT02045212), also has a primary goal of assessing short-term safety, but differs from the Stage 1 study by using the compound in humans with the disease of interest and having a placebo comparison. RPh201 is currently being studied in a Phase 3 trial (clinicaltrials.gov NCT03547206), which is also an RCT, with a primary outcome of efficacy. RANDOMIZED CONTROLLED TRIALS AS A CLINICAL STUDY DESIGN OPTION The purpose of RCTs is to evaluate whether an intervention results in a net improvement in average health. It is an experimental study design that aims to determine the effect of an intervention (independent variable) on efficacy and safety outcomes (dependent variables) while controlling for potential confounding variables (controlled variables). The salient features are prospective design, comparison between 2 or more treatment groups to address bias introduced by differential care (placebo effect), and randomization to treatment to address selection bias introduced by nonrandomized treatment allocation. Randomization also helps to balance for known and unknown confounding variables between groups. A common feature is blinding/masking of subjects, examiners, or both. Rath et al’s study has all of these features being prospective with randomized treatment allocation and masking of everyone except the pharmacist. However, these features alone do not define the quality of the study. Oxford Centre for Evidence Based Medicine ranks RCT as the pinnacle of single study designs with regards to providing evidence of treatment benefit and harms, superior to nonrandomized controlled cohort or follow-up studies, which have potential for selection bias (Level 3), case series, case control or historically controlled studies, which have the potential for differential care bias (Level 4) and mechanism-based reasoning (Level 5). In contrast to RCT, these other study types rely on statistical approaches rather than randomization for addressing known confounding variables. Despite being Departments of Ophthalmology (HEM) and Neurology and Neurological Sciences (HEM), Stanford University, Palo Alto, California; Department of Statistical Science (JC), Southern Methodist University, University Park, Dallas, Texas; and Department of Ophthalmology (SLP), University of California at Los Angeles, Los Angeles, California. Unrestricted grant from Research to Prevent Blindness to Stanford University Department of Ophthalmology. The authors report no conflicts of interest. Address correspondence to Heather E. Moss, MD, PhD, Spencer Center for Vision Research, Stanford Department of Ophthalmology, 2370 Watson Court, Suite 200, MS 5353, Palo Alto, CA 94303; E-mail: [e-mail] Moss et al: J Neuro-Ophthalmol 2020; 40: 3-7 3 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial classified as lower levels of evidence, it is important to note that other treatment study designs play an important scientific role in generating the evidence needed to design RCTs, or providing best available evidence when an RCT is not practical. Furthermore, there are many clinical research questions for which an RCT is not appropriate, including questions of epidemiology, accuracy of diagnostic tests and prognosis and risk. RANDOMIZED CONTROLLED TRIALS STUDY DESIGN CONSIDERATIONS Study Question An RCT may be an appropriate study design for clinical questions regarding the effect of an intervention in a group of people. The intervention must be specific so that it can be administered in a standardized fashion to subjects in the trial. There must be at least one comparison intervention such as placebo, and selection of this comparison should take into account potential for masking participants and study personnel. The group of people are often characterized by a disease, although this is not necessarily the case (e.g., prevention trials). The study question must be ethical, such that it compares interventions that are equivalent or better to standard of care. There must be clinical equipoise between the interventions, meaning that there is genuine uncertainty regarding which treatment is better. In the case of Rath et al’s trial, the standard of care for chronic nonarteritic anterior ischemic optic (NAION) is no treatment, which makes a placebo an appropriate comparison treatment. Based on Phase I trial safety data, there is equipoise regarding benefits and risks of RPh201 compared with placebo in the chronic NAION population. Practical considerations may also shape the study question and whether or not an RCT is an appropriate study design. Subject Selection Inherent in the study question is the general group of subjects being studied. Detailed inclusion and exclusion criteria are necessary to operationalize how the subjects will be identified. The primary goal is to include subjects in whom there is high diagnostic certainty regarding the disease of interest and in whom endpoints can be assessed. Typically subjects who have other medical conditions that may affect study outcomes, who may not be able to complete the study (e.g., because of an allergy to the intervention or potential pregnancy in a study with an intervention with unknown teratogenicity) or in whom outcome measurements may have poor precision, reliability or no room for improvement are excluded. For example, Rath et al’s inclusion criteria required sufficient visual impairment so that improvement could be measured (visual acuity [VA] , 20/200 or visual field , 15°). However, it was noted after study completion that subjects with severe 4 vision loss (“off chart” VA) had poor reliability regarding outcome measurement than those with “on chart” VA, which was addressed through a post-hoc analysis. Ultimately, subject selection must balance practicality and generalizability. For the clinical trialist, having a restricted homogeneous group of subjects may ease study design, analysis, and interpretation, but cause recruitment challenges. For the practicing physician and critical reader of the medical literature, a homogeneous sample can limit generalizability to their patient population. Endpoints Endpoints are the practical details of how positive and negative effects of the treatment are measured. They need to be clinically relevant, or surrogates for something clinically relevant, and measurable, with the expectation that change can be detected during the study time frame. Ideally, they are objective, although subjective outcomes are often also included such as for assessment of subject symptoms or adverse events. For diseases affecting the afferent visual pathway, such as NAION, VA and automated perimetry are common, clinically relevant RCT outcomes, although their inherently subjective nature and variability must be taken to account. Rath et al reinforce this point in their observation of initial improvement in VA in both treatment groups, suggesting regression to the mean and a learning effect, which they addressed in a post-hoc analysis by averaging VA measures from early visits to serve as the baseline. Because of their prospective nature, RCTs offer the opportunity for detailed standardized data collection. However, perfect and comprehensive data is expensive both financially and regarding subject time, which may in turn limit subject recruitment and retention. Therefore, many trials leverage “free” data collected as part of clinical care and some are based almost entirely in established patient registries (5). Masking (Blinding) Masking (blinding) is an important feature of many RCTs to reduce ascertainment bias, wherein measurement or report of outcomes is affected by subconscious or conscious favoring of a treatment. Subjective outcomes are especially prone to this. To accomplish masking, interventions need to be designed so that they are indistinguishable to whomever is masked (participants and/or examiners). By necessity, there are some study personnel who are unmasked. In Rath et al’s trial the pharmacist who dispensed the treatment was the only unmasked individual. In more complicated trials, particularly those involving a sham procedure, additional unmasked team members may be necessary. Randomization Randomization prevents biased assignment of patients to groups by the treating physician and thereby reduces Moss et al: J Neuro-Ophthalmol 2020; 40: 3-7 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial selection bias. It is also an important tool for distributing unknown and known confounding variables between study groups. Rath et al used a straight forward randomization procedure with the order of assignment established before study initiation. More complex strategies include those to address equal allocation of confounding variables (stratified), or equal allocation to treatments over time (block). Other strategies simplify study administration such as group assignment to intervention (cluster) or adaptive designs that allow for stopping at various points in the trial (group sequential design). For example, in cluster randomization randomized by physicians, all patients cared for by a physician receive the same treatment as assigned to that physician. Advantages of cluster randomization design include administrative convenience and avoiding treatment contamination. However, this brings analytical challenges. Sample Size One key task at the design stage is to determine the total number of subjects to enroll in the trial so that the probability (power) of detecting a clinically meaningful treatment effect is adequately large. This requires a target effect size, which takes into account expected difference between groups and variability within groups, and can be specified based on previous studies or clinical data. For example, for a continuous endpoint and 2 groups, effect size is the quotient of the difference in group means and the pooled standard deviation. It is critical that sample size calculation should be consistent with the study design (e.g., individual randomization, cluster randomization, or group sequential design) and the planned analysis for the primary endpoint (e.g., continuous or binary outcomes). Rath et al’s trial is a Phase 2a safety and efficacy study, which is a proof-of-concept trial and was not powered to detect small or moderate effects, but rather to assess the safety of RPh201 and to explore potential efficacy signals with RPh201 compared with vehicle control. In such proof-of-concept trials, sample size calculation is not strictly required. Analytical Principle The standard practice in the analysis of randomized trials is to follow the principle of “intention to treat” (ITT). That is, all randomized participants are analyzed according to the group they were originally assigned, regardless of what treatment (if any) they received. Intention to treat analysis guarantees that the groups being compared have similar characteristics as achieved through randomization. In addition, it usually best reflects the effects of treatment in everyday practice. The estimated treatment effect, however, is generally conservative. Alternatively, in “per protocol” analysis, participants are analyzed according to the treatments they actually received. It can introduce attrition bias, in which the groups of patients being compared no longer Moss et al: J Neuro-Ophthalmol 2020; 40: 3-7 have similar characteristics. The results of per protocol analysis usually provide a lower level of evidence, but better reflect the effects of treatment. Rath et al followed the intention-to-treat principle in their data analysis. Data Analysis Depending on the experimental design and the types of outcomes, different analytical approaches should be used. For example, in Rath et al’s trial, the t test to compare the continuous outcome best-corrected VA between the 2 groups is appropriate, because the participants are individually randomized to ensure even distribution of confounding covariates. In practice, the clustered randomization design has been widely used, where the randomization unit is cluster of participants (e.g., treated by the same physicians or in the same clinics). This brings an analytical challenge because of intracluster correlation existing among measurements obtained from patients within a cluster. More sophisticated approaches, such as generalized estimating equation or mixed effect models, have been developed to account for the intracluster correlation. In larger randomized trials (such as Phase III confirmatory trials), the group sequential design has been frequently used. It includes a predetermined number of stages: interim stages and the final stage. Specifications at each stage include sample size, critical values, and stopping criterion to either support or reject the null hypothesis. During the trial, interim analysis is performed at each stage, which involves calculating a test statistic and comparing it to the critical values to decide whether to continue or terminate the trial. The group sequential design offers researchers the flexibility to stop a trial early based on overwhelming evidence of efficacy or futility. It can therefore save time and resources, and reduce the exposure of patients to inferior treatments. Analyzing the data multiple times, however, leads to inflation of Type I error. The alpha spending function approach (6) has been widely used to adjust the critical values at each stage, such that the overall Type 1 error rate remains at the desired level. In practice, pragmatic issues may occur, and how these will be approached or avoided is best considered during the study design phase. Interim analyses regarding safety monitoring and prespecification of rules for stopping the trial based on these should be prespecified. Addressing imperfections in the trial (e.g., uneven distribution of confounding variables between groups) or data (e.g., missing data) can be minimized through careful study design and administration and addressed analytically using a variety of techniques. For example, the role of a mediating variable in the cause–effect pathway can be investigated using mediation analysis. Approaches to missing data and mediation analysis were both applied in the idiopathic intracranial hypertension treatment trial (7). 5 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial Protocol The product of careful study design is the study protocol and manual of procedures, which operationalize the study in “how to” manuals. The Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement is a checklist and figure to guide assembly of a comprehensive trial protocol (available at https://www.spirit-statement.org) (8). RANDOMIZED CONTROLLED TRIALS STUDY ADMINISTRATION CONSIDERATIONS The logistics of administering an RCT are substantial and increase multifold for multisite compared with single-site studies. However, multisite studies are often desirable to achieve adequate sample size and increase generalizability. An administrative structure with clearly defined roles is important for overall study management and for each site. Established workflows for recording and checking data in line with privacy and confidentiality regulations are necessary. As with other clinical research studies, compliance with human subjects’ regulations is required including approval from an institutional review board with appropriate jurisdiction. Trial Registration Many regulators (e.g., the United States Food and Drug Administration), funders (e.g., the United States National Institutes of Health) and journals require registration of clinical trials in a public database such as clinicaltrials.gov. Beyond meeting a regulatory requirement there are other benefits of posting a planned study and result to a range of stakeholders in clinical research (9,10). In the case of RPh 201, both Rath et al’s study and a Phase 3 study are currently listed on clinicaltrials.gov (NCT02045212, NCT03547206). It is important for users to recognize that inclusion of a clinical study in a public database is not an endorsement of the quality of the study question or design. Recruitment and Retention of Subjects Recruitment and retention of subjects is of great importance, because they are the basis of the trial. Although not part of the formal study design, study design decisions must take into account their impacts of subject recruitment and retention. Protocol Changes Protocol changes are inevitable in even the best planned study in response to information gained during the course of the trial. Most common is changes to subject selection criteria, for example, Rath et al excluded traumatic optic neuropathy when no subjects had been recruited with this diagnosis. Subject selection criteria may also be altered to address newly identified barriers to recruitment. Study procedures and endpoint measurement may be altered to address challenges in subject retention. 6 POST-HOC ANALYSIS OF RANDOMIZED CONTROLLED TRIALS DATA Inherent in the experimental design of an RCT is the study question and the analysis approach that will be used to answer it. Randomized controlled trials are not designed for new questions that arise during study administration or after completion of the trial. Even though these new questions may be answered using data that were prospectively collected in an RCT, they cannot be considered to have been answered using an RCT study design. Though they can generate important insights, these post-hoc analyses must be considered hypothesis generating. In the case of Rath et al, post-hoc analyses were performed with a new definition of baseline VA (averaged over initial visits) and within the “on chart” VA subgroup. Though these cannot be considered results of the prospective experimental trial, they are important for the design of subsequent trials, including the Phase 3 trial of Rph201 that is underway. REPORTING RANDOMIZED CONTROLLED TRIALS Critical reading of RCT has been made easier since the publication of the Consolidated Standard of Reporting Trials (CONSORT) statement in 2010 (11). This consists of a checklist of 25 main items, 12 subitems, and a standard flow diagram to indicate subject enrollment and follow-up (available at http://www.consort-statement.org). It is not a tool for evaluating study quality, rather it ensures that sufficient information is included in the manuscript with which to evaluate study quality. Rath et al’s figure 1 is the CONSORT flow diagram and their manuscript addresses the checklist items, although some are included as supplementary on-line data. READING RANDOMIZED CONTROLLED TRIALS AND APPLYING THEM TO PRACTICE For a practitioner reading a report of an RCT, the key question is generalizability to their practice. All aspects of the trial including the study question, subject selection, and endpoints should be evaluated through this lens to determine whether the results are applicable to their patients. Alhough RCTs are designed to minimize bias affecting study results, there is inherent bias in the study design that needs to be considered. In particular, what and who was not studied is as important to generalizability as what or who was studied. For example, in Rath et al’s study 22/23 screened subjects were enrolled in the study, but we do not know whether this is a biased sample of the chronic NAION population. Such bias could have occurred through which potential subjects were offered screening and which potential subjects declined screening. The reader Moss et al: J Neuro-Ophthalmol 2020; 40: 3-7 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial should also be mindful of bias introduced by subject drop out and missing data. Generalizability of an RCT may be limited further, because the clinical attention provided to subjects in the trial is likely much more than that provided to patients in nontrial clinical settings. REFERENCES 1. The Oxford Levels of Evidence 2. Available at: https://www.cebm.net/index.aspx?o=5653. Accessed July 13, 2019. 2. The 2011 Oxford CEBM Levels of Evidence [Introductory Document]. Available at: https://www.cebm.net/index.aspx?o=5653. Accessed July 13, 2019. 3. Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence [Background Document]. Available at: https://www.cebm.net/index.aspx?o=5653. Accessed July 13, 2019. 4. Rath EZ, Hazan Z, Adamsky K, Solomon A, Segal ZI, Levin LA. Randomized controlled phase 2a study of RPh201 in previous nonarteritic anterior ischemic optic neuropathy. J Neuroophthalmol 2019;39:291–298. Moss et al: J Neuro-Ophthalmol 2020; 40: 3-7 5. Li G, Sajobi TT, Menon BK, Korngut L, Lowerison M, James M, Wilton SB, Williamson T, Gill S, Drogos LL, Smith EE, Vohra S, Hill MD, Thabane L. Registry-based randomized controlled trials- what are the advantages, challenges, and areas for future research? J Clin Epidemiol. 2016;80:16–24. 6. DeMets DL, Lan KK. Interim analysis: the alpha spending function approach. Stat Med. 1994;13:1341–1352; discussion 1353–1346. 7. Wall M, McDermott MP, Kieburtz KD, Corbett JJ, Feldon SE, Friedman DI, Katz DM, Keltner JL, Schron EB, Kupersmith MJ. Effect of acetazolamide on visual function in patients with idiopathic intracranial hypertension and mild visual loss: the idiopathic intracranial hypertension treatment trial. JAMA. 2014;311:1641–1651. 8. Chan AW, Tetzlaff JM, Altman DG, Dickersin K, Moher D. SPIRIT 2013: new guidance for content of clinical trial protocols. Lancet. 2013;381:91–92. 9. Zarin DA, Keselman A. Registering a clinical trial in ClinicalTrials.gov. Chest. 2007;131:909–912. 10. Tse T, Williams RJ, Zarin DA. Reporting “basic results” in ClinicalTrials.gov. Chest. 2009;136:295–303. 11. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332. 7 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited.