Mitigating Study Risk With Performance Analytics

Increasingly, both sponsors and research sites are turning to various types of performance metrics to support risk-based management of clinical trials. But collecting and analyzing performance data after a trial has begun may not be the best approach; if data on site and researcher performance can be collected and analyzed while they are being trained on a protocol, for instance, sponsors have an opportunity to head off key risks before the first patient is enrolled in a study.

The research industry has been paying closer attention to use of performance analytics to improve the way trials are conducted. In 2017, for instance, ACRP launched an initiative with CRO Analytics to provide insight on performance analytics and how they might improve clinical research, including reductions in costs and shorter timelines.

Avoiding problems that put data at risk or cause adverse events, which could undermine a study; this requires vigorous risk management, or “the exercise of thinking in advance about study risks and implementing mitigation strategies,” Dawn Niccum, executive director of QA and compliance at inSeption Group, said in a 2021 issue of Bioprocess Online. “It demands stakeholders identify critical study processes and calculate risk associated with those processes.”

And in a 2020 issue of Clinical Researcher, Patrick Hughes, co-founder and chief commercial officer at CluePoints, noted that the ICH guideline on GCPs extends risk-based management to all aspects of clinical trial execution, an approach that “has opened a tremendous opportunity to plan and manage clinical research more effectively and efficiently.” The ICH E6(R2) guideline on GCPs outlines what an ideal risk-based quality management system should include, such as risk identification, evaluation, control and communication, among other features.

“The first step in proactive data monitoring is to identify what is possible to mitigate, eliminate, and accept,” Hughes wrote. “This all forms part of various plans, including those for data, training, monitoring, statistical analysis, safety, medical monitoring, quality, and other functional plans.”

In other words, performance analytics can help reduce risks and improve clinical research. But when are the metrics gathered that form the basis of those analytics? Ideally, sponsors would want to have that information as early in the process as possible. And protocol-specific training could offer the perfect opportunity to gather data on where critical problems—those that can impact data quality and patient safety—are most likely to occur.

And the earlier sponsors can get quality information on what is likely to happen at each site participating in an upcoming clinical trial, the more likely they will be to get everything right from the start. Getting that information up-front is a huge risk management effort.

The value of training-generated analytics

Better use of analytics to measure and track how individual sites, as well as specific staff, are performing in terms of properly following the protocol can help sponsors better target their remediation efforts. Necessary retraining could be provided only at those sites and to those individuals that are having deviations, for instance. And that training could focus specifically on the deviations seen.

Furthermore, sponsors often may provide remedial training to all staff at all sites rather than targeting only the sites—or the individuals—making the errors.

And that’s not all. Sponsors can also find and address these risks during initial training if they plan well. Providing training that allows research staff to practice their roles and tasks under a given protocol before a study begins can allow sponsors to collect clear data on how well-prepared each site and its staff is to conduct the protocol correctly.

The right training can help identify these areas and better orient both sponsors and sites to mitigate these risks. The key is to identify metrics that will help identify where risks are more likely to occur and track the occurrence of risky actions and decisions by clinical research staff.

And among the primary risks that sponsors seek to avoid are protocol deviations, which can have significant impacts on clinical trials. The FDA reported in its annual BIMO metrics report covering 2021 inspectional findings that protocol deviations or failure to follow the investigational plan continue to hold the top spot among most-frequent Form 483 observations, a position it held in 2020 and for the last several years.

When crafting a protocol, sponsors generally take a prescriptive view of where they think problems may be likely to occur. For instance, procedures required under the protocol that differ from the usual standard of care could pose a risk of deviations due to physicians, nurses and other staff falling automatically into the more typical way they are used to doing things.

Performance metrics tracked and collected during initial training can produce analytics that allow sites and sponsors to identify and correct problems before it affects study participants, thus reducing the risk of protocol deviations, as well as risks that could directly affect patient health or data integrity. 

But traditional forms of training don’t provide a way to measure how well various research staff are able to perform their jobs under a given protocol. They don’t let sponsors see if risk has been mitigated until they start seeing deviations after a study has started. When that occurs, sponsors must engage in those costly and time-intensive remediation efforts. In addition to taking up sponsor time and money, these retraining efforts also can eat up a lot of site time, often without additional payment.

Simulation-based training, on the other hand, lends itself especially well to collecting and presenting this type of data. Pro-ficiency, for example, provides simulation-based training that lets sites practice all parts of a protocol in a consequence-free environment. Every decision users make during a simulation experience is tracked, collected, and presented in actionable performance and compliance reports. Moreover, these reports can provide valuable insights to sponsors, with the ability to identify which sites and which individuals are having issues with a particular part of the protocol before they affect study timelines or budgets.

This heat map dashboard is just one example of the reporting capabilities simulation-based training offers. From sites to individual investigators, these dashboards can be invaluable in promptly detecting problematic areas and providing proactive assistance before lagging performance leads to deviations.

Simulation-based training customized for a given study, for instance, can track performance at both the individual and site level. These analytics can help sponsors better identify risk areas related to deviation from the protocol and address them with targeted training before the first patient is enrolled.

Behavior-based performance metrics can help sponsors predict performance by sites and their staff. And these metrics, or analytics, can be used to identify weak sites, under-performing staff, or even potential pain points in the protocol itself. Sponsors can take this information and provide carefully targeted support to the individuals or sites that need it most, heading off risks of protocol deviations before they occur within the actual study.

Pretesting of staff decisions

This type of training system can be used to determine staff’s first responses to instructions on how to conduct protocol procedures, and to make corrections if their first actions are incorrect.

When determining what metrics to track, sponsors should think through the elements of a study that will most impact its success and what the greatest impact of noncompliance could be. For instance, deviations like lack of an initial or signature on some paperwork are easy to fix, while deviations in dosing or patient procedures would cause more problems. Important considerations are whether lab techs and other staff are handling the investigational product correctly and if patient visits and tests are conducted correctly.

The analytics can help sponsors see what sites’ natural inclinations lie in areas of risk. For instance, is there a particular procedure that they tend to miss the mark on? This will help sponsors drill down to how staff are handling their protocol-specific roles and allow for highly targeted remedial work, if any is needed.

This information allows sponsors to tell which sites or coordinators struggle with correctly applying inclusion/exclusion criteria when enrolling patients, or which sites have problems with the dosing regimen under the protocol. Sponsors can then address each site’s specific weaknesses to uncover the root of the problem and provide additional training to avoid problems when the study begins.

For instance, a protocol might include a complicated preparation procedure for the investigational drug that states the product can’t be shaken, but must be undulated to mix – and if bubbles are seen, it cannot be used. If a researcher’s first response to practicing the procedure is to shake the product, see bubbles and set it down, that can indicate to the sponsor that this staff member did not fully understand the prep instructions. 

Analytics can help point out weak areas in a protocol. In the example above, if the investigational product prep procedure is a challenge at multiple sites, that could be a signal to the sponsor that extra effort is needed in this part of the training before the trial starts.

In short, customized simulation-based training that measures carefully selected performance metrics can provide analytics that allow sponsors to identify key areas of risks, particularly regarding protocol deviations that could impact patient safety or data quality and integrity. Not only does this reduce the risk of citations for protocol deviations once a study is underway, it also can save time and money that would be spent on re-training when deviations began occurring with real study participants.

Analytics allows for both sponsors and sites to understand the problem and make better plans to fix them, taking less time away from the clinical trial and reducing overall study risk.

To learn more about a training approach which integrates predictive analytics, visit http://proficiency1.wpenginepowered.com/prescriptive-analytics.

Jenna Rouse

As Chief Experience Officer at Pro-ficiency, Jenna helps clients realize the full potential of their partnership with Pro-ficiency. Jenna has spent 16 years at the Association of Clinical Research Professionals (ACRP), including serving in various roles such as Business Development leader. Throughout her time at ACRP she helped hundreds of companies integrate competency-based learning, hosted by Pro-ficiency, into their workforce initiatives. In this work, Jenna began noticing her clients’ appreciation for the incredible Pro-ficiency platform, simulations, and customer service. Now, Jenna helps clients (including ACRP) realize the full potential of their partnership with Pro-ficiency.