Preventing Protocol Deviations With Simulation-Based Training

Protocol deviations can have a devastating impact on the quality and integrity of key study data and potentially affect patient safety. And there can be serious regulatory repercussions for failing to follow the protocol precisely, as well. Taking the right approach when training investigators, research coordinators and other key staff can be critical to avoiding protocol deviations.

And simulation-based training in particular allows research staff to practice a protocol and all its nuances in advance of applying it to real patients and real data, with the associated real potential risks. And in addition to identifying and addressing protocol deviation risks, the designed scenarios can allow research staff to learn how to prevent the issues that lead to deviations before they occur.

FDA inspectors will focus primarily on protocol deviations that are preventable and that the agency deems important, meaning the deviation may significantly impact the completeness, accuracy and/or reliability of key study data or may significantly affect patient rights or safety.

The FDA’s report on inspection trends for 2020 indicated that failure to follow the study protocol remains the most-frequent Form 483 observation made during BIMO inspections of clinical research operations. Some of the top issues mentioned have included patients enrolled who did not meet inclusion criteria or had an exclusion, incorrect performance of study procedures, missed labs or assessments, missing or late AE/SAE reports and missing protocol-required documentation. Other top observations were:

• Inaccurate or inadequate case histories;
• Insufficient accountability records;
• IRB problems;
• Failure to report AEs to sponsors; and
• Problems with informed consent forms or procedures.

And while warning letters issued to clinical investigations are not commonplace, that is also a risk that can derail a study by diverting resources to correct problems noted by the FDA so as to keep the research moving forward. For instance, the FDA has issued four warning letters to clinical investigators in 2021 to date, three of which have included “failure to ensure that the investigation was conducted according to the investigational plan” among the citations. Specific violations included:

• Enrolling patients that did not meet all inclusion criteria;
• Enrolling patients that showed exclusions at the initial screening visit;
• Not applying defined patient randomization criteria and procedures;
• Failure to perform required lab tests at time points specified in the protocol; and
• Failure to follow the protocol-specified dosing regimens.

The FDA noted that these trends have been consistent for the last several years, with failure to follow the protocol or investigational plan consistently landing in first place for most-common Form 483 observations. The reasons for protocol deviations are many’, Beth Harper, chief learning officer at Pro-ficiency, a company that provides customized training solutions for clinical research sites, said. And appropriate training can help to alleviate some of the issues that lead to protocol deviations.

Complex protocols challenge protocol compliance

Protocol complexity is one factor that could contribute to protocol deviations. In early 2021, the Tufts Center for the Study of Drug Development (CSDD) noted that protocols have been steadily increasing in complexity since 2009. For instance, the CSDD report said that Phase II and III protocols now generally have about 20 endpoints, with an average of 1.6 primary endpoints, up 27% since 2009. Additionally, the mean number of distinct protocol procedures has risen 44% in the same time frame.

And this increase in procedures and endpoints could provide more opportunities for misunderstanding or mistakes.

“As protocols become more complex, it is possible that some procedures may not be feasible within the timelines specified, for instance, because operational needs were not considered alongside scientific requirements,” Harper said.

And traditional training approaches may not be sufficient to the task of ensuring that research staff fully understand the fine points of all procedures included in a protocol. Simulation-based training that requires researchers to go through individual steps of a process or procedure in a life-like scenario can be a more effective way of preparing investigators, coordinators and other research staff to avoid protocol deviations.

The value of this type of training is gaining attention throughout the industry. For instance, Elio Mazzone et al. noted in a paper in the August 2021 Annals of Surgery that competency-based progression simulation training was more effective than traditional training in reducing procedural errors and increasing the number of correct steps taken. A key, the paper said, lies in continuous feedback and the ability to repeat tasks until they are done correctly.

Pro-ficiency’s training solutions, for instance, provide realistic scenarios that give learners freedom to make a variety of decisions. After considering the range of possibilities for a procedure, for instance, the training will guide research staff through likely scenarios, offering multiple options at key decision points and providing immediate feedback as to whether the learner is on the right track or not.

Decision-making skills emphasized

And as the research staff progresses through each procedure, from enrollment and consenting through schedule lab tests and exams, Pro-ficiency’s training system tracks each decision made, provides immediate feedback and generates behavior-based analytics that can flag areas where staff may need additional training or where the protocol may be confusing and need clarification.

Additionally, Harper noted, Pro-ficiency experts dissect each protocol to predict potential performance problems, looking for areas where researchers may be tripped up.

During training utilizing simulation methodologies, however, research staff actively participate in making the types of decisions necessary to ensure that the protocol is followed precisely. Simulation-based training that walks researchers through the necessary steps for all processes and procedures required by a protocol can help ensure that they fully understand the protocol.

It can also help prepare research staff to correctly handle unusual situations or grey areas that might arise.

In some cases, research staff may need to go through a scenario several times before getting to the right answer. In those instances, it’s important to be able to determine whether there is a problem with an individual’s understanding, the level of knowledge and experience at a given site or, if multiple employees at multiple sites have similar difficulties, whether the protocol itself is unclear.

And in other cases, a protocol amendment may be required. Alternately, supplemental training and job aids, such as checklists, cheat sheets or decision trees, may be provided to help ensure that researchers conduct all activities in line with the protocol.

It’s also important to remember that there may be considerable lag time—months in the case of rare diseases or specific types of cancer, for instance—between researchers receiving training; job aids can be useful for these instances, as well.

Other factors that might lead to protocol deviations include inexperienced site staff, which also can be addressed via training designed to target weak areas. Simulated scenarios can be developed that will guide research staff through procedures or in using equipment with which they might be unfamiliar.

Other potential factors that could lead to protocol deviations include language barriers, regional availability of supplies, equipment or staff expertise and issues of cultural sensitivity. For instance, CSDD noted in its 2021 report that the mean number of countries in which studies are conducted has also grown substantially since 2009.

As more trials include multinational sites, this can be an important consideration. Voiceovers and subtitles can address language differences, and specialized training modules may be designed for sites facing availability or cultural issues.

David Hadden

Dave Hadden is an entrepreneur and technology innovator, founding Pro-ficiency and pioneering the fields of A.I.-based medical decision-support, Training Analytics, and Virtual Patient Simulation (VPS). Dave has focused his passion for technological innovation and learning systems in the field of clinical trials, helping sponsors make their studies more accurate and efficient through finding the right technology mix such as virtualization, performance management and applied behavioral sciences to produce the most effective, lasting, and engaging results for clinical trials.