Reducing Protocol Deviations With Training-Based Metrics

Protocol deviations can have a detrimental effect on new drug development, adding to costs and time to clinical trials and even putting the acceptability of data generated during a study at risk. Despite this, protocol deviations seem to be viewed as just part of doing business in the clinical research arena. And one question that arises is whether training in individual protocols is adequate, given the frequency with which deviations occur.

In its recent analysis of inspectional findings during 2021, the FDA said in its annual BIMO metrics report that protocol deviations or failure to follow the investigational plan was the most common observation listed on Form 483s after a BIMO inspection. This was also the most-frequent Form 483 observation during 2020, and has been at the top of the list for the last several years, the FDA noted.

And in its January/February 2022 Impact Report, Tufts University Center for the Study of Drug Development (CSDD) reported that protocol deviations rose in the 2018-2020 period compared to the 2013-2015 period, as have substantial amendments to protocols. According to the CSDD report, Phase 3 trials had the highest number of deviations per protocol, with an average of 118.5, which affected 32.8% of patients enrolled. Phase 2 studies saw an average of 75.3 deviations affecting 30% of patients, and Phase 1 trials had an average of just 8.7 deviations affecting 15.3% of patients.

Oncology trials saw a higher number of deviations, on average, compared to other areas of research, with Phase 2 and 3 oncology protocols having 30% more deviations compared to non-oncology protocols. Additionally, protocols outsourced to CROs had more deviations and substantial amendments per protocol, CSDD reported.

But despite the potential for negative repercussions, protocol deviations remain an ongoing challenge for researchers. As a matter of fact, in a June 2020 blog post, clinical trial platform provider Castor referred to protocol deviations as “the new normal.”

Precisely quantifying the impact of all of these protocol deviations on clinical research is difficult because of the variation among different areas of study and even among individual clinical trials. However, it is clear that deviations do affect trial progress and outcomes.

Industry seeking to change

Despite these figures, however, researchers and sponsors appear dedicated to reducing or even eliminating protocol deviations. In fact, some research organizations use protocol compliance as one of the metrics by which site performance is measured. For instance, Diane Whitham et al., on behalf of the Site Performance Metrics for Multicenter Randomized Trials Collaboration, listed several metrics related to protocol compliance in an October 2018 Trials article, including:

• Percentage of randomized participants with at least one protocol violation;
• Percentage of randomized participants receiving allocated intervention as intended per protocol;
• Number of missed visits per number of randomized participants;
• Number of late visits per number of randomized participants; and
• Number of critical or major audit findings per number of randomized participants.
• And the entirety of a seasoned clinical research audience attending a 2021 webinar agreed that one key metric should define success for site training: zero important protocol deviations.

The message seems to be that the goal of zero deviations—or at least zero important deviations—can be achieved with the right strategy. The industry is clearly keen to avoid the potential blowback from protocol deviations, which can include additional time to complete a trial, the costs associated with extending a trial.

Protocol deviations can increase costs due to the study having to be extended, additional patients enrolled, replicated work and prolonged timeline for product development. Additional monitoring and retraining costs may be incurred, as well. Deviations also may have an impact on data quality that could ultimately have long-term implications on product approval.

In fact, the ICH E9 guidance, Statistical Principles for Clinical Trials, stated clearly that protocol deviations “always bear the potential of invalidating the trial results.”

And all protocol deviations require documentation and reporting, Laurie Halloran, president and CEO of Halloran Consulting Group, noted in the May 2, 2018 Life Science Leader, activities that eat up staff resources and can add to costs in that way. If an unreported deviation is discovered during a regulatory inspection, it is likely to result in penalties from the FDA or other regulators.

In some cases, deviations may lead to protocol amendments, which can delay trial completion, potentially costing $35,000 per day or more, according to a 2015 survey by Clinverse, a statistic still used for cost estimates. Amendments can add an average of 30 days—and associated costs—to the length of a study, according to a 2021 report from CSDD.

Protocol deviations—whether enrolling an unqualified patient, incorrectly performing a visit or procedure or failing to collect data necessary to interpret primary endpoints—can result in unusable data that requires additional time and resources to replicate. In its March/April 2021 Impact Report, CSDD noted that mid-study amendments typically require 30 days to complete before a study can resume, adding significantly to the time required to complete a clinical trial.

Addressing deviation risks with training

There are myriad reasons behind protocol deviations. Ever-increasing protocol complexity could be one factor that contributes to protocol deviations. CSDD noted that protocols have been steadily increasing in complexity since 2009, with increasing numbers of endpoints and distinct protocol procedures. Those increases provide more opportunities for investigators or research staff to make mistakes.

Sometimes a change in procedure may be made to ensure patient safety, as in the case of emergency treatment, Halloran said.

And social distancing requirements during the COVID-19 pandemic spurred some protocol deviations, such as replacing in-person site visits for patients with remote or telehealth visits. The FDA in August 2021 updated its guidance, Conduct of Clinical Trials of Medical Products During the COVID-19 Public Health Emergency, recognizing these necessities and clarifying what protocol deviations due to COVID would be acceptable with appropriate reporting.

But these relatively innocuous reasons are not the only causes of protocol deviations. For instance, deviations may occur because the protocol is hard to understand or allows different interpretations at different sites. And they may occur because study teams have not been trained adequately or because knowledge isn’t transferred well during staff turnover. In addition, Insufficient performance-based tracking throughout site training makes it difficult for proactive actions to be taken to address high-risk areas for protocol deviations that may result from site error.

All of these causes can be addressed with the right type of training. Thorough, protocol-specific training can help ensure that investigators, coordinators and other research staff fully understand how to perform all tasks included in the protocol. And properly designed training will develop mental muscle memory so that researchers can perform correctly under pressure or when substantial time has passed since training. Besides developing participants’ comprehension, proper training should track performance and allow sponsors to identify and correct potential problems before they impact a study’s budget or timeline.

For instance, protocol simulation-based training can mimic realistic scenarios and guide research staff through each step, including making the right decision when problems occur, like:

• A patient takes a restricted medication just before the randomization visit;
• A temperature excursion occurs for the investigational product; or
• A patient misses a dose or a visit where a procedure was to be completed.

Ideally, the training allows the learner to critically think through preventive actions. The simulation scenarios should not only model what can go wrong and assess potential corrective actions, but more importantly, how the issues can be prevented. When users make decisions in a simulation experience it can be tracked, collected, and presented in performance and compliance reports. These reports provide invaluable insight that can identify and address individual or site-wide weak points before a protocol deviation occurs. In contrast to traditional training, simulation-based training aims to ensure study training is consistent, intense, and proactive.

Simulation-based training in particular allows research staff to practice a protocol and all its nuances in advance of applying it to real patients and real data, with the associated real potential risks. And in addition to identifying and addressing protocol deviation risks, the designed scenarios can allow research staff to learn how to prevent the issues that lead to deviations before they arise.

And this practical approach to training can help sponsors and researchers reduce the incidence of protocol deviations and avoid the additional costs—in both time and money—associated with fixing those deviations.

Contact us to discover how our simulation-based training approach can reduce protocol deviations across studies.

Beth Harper

Beth Harper, Chief Learning Officer at Pro-ficiency, is on a mission to eliminate protocol deviations. Beth brings 35-plus years of experience in clinical research operations to Pro-ficiency, utilizing this knowledge along with her years of creating live protocol simulation training to create cutting-edge simulation training programs at Pro-ficiency. Beth has published and presented extensively in the areas of study feasibility, site selection, patient recruitment, protocol optimization, sponsor/site relationship management and clinical research professional competencies. Beth received her B. S. in Occupational Therapy from the University of Wisconsin and an M.B.A. from the University of Texas.