Enhancing Surgical Quality Assurance: Development and Reliability of a Competency Assessment Tool for Magnetic Sphincter Augmentation

Highlight

This article details the creation and validation of a video-based competency assessment tool (MSA CAT) specifically developed for magnetic sphincter augmentation (MSA) surgery. The tool was developed through a rigorous Delphi consensus process involving international experts and demonstrated good inter- and intra-assessor reliability. It forms a key part of the surgical quality assurance program in the multicenter, international GOLF trial comparing MSA with fundoplication for gastroesophageal reflux disease.

Study Background

Gastroesophageal reflux disease (GERD) often requires surgical intervention when conservative management fails. Magnetic sphincter augmentation (MSA) is an innovative minimally invasive surgical alternative to traditional fundoplication. However, ensuring consistent, high-quality surgical performance across centers and surgeons remains challenging. Surgical quality assurance (SQA) programs aim to objectively evaluate and maintain operative standards by measuring adherence to standardized operative steps, identifying technical errors, and grading proficiency.

The GOLF trial (ISRCTN: 1353317) is the first international, multicenter randomized controlled trial (RCT) comparing MSA with fundoplication, incorporating a novel video-based SQA program. Given the technical complexities of MSA and its relative novelty, developing a robust competency assessment tool to monitor surgical quality is essential to trial integrity and optimal patient outcomes.

Study Design

To develop the MSA competency assessment tool (MSA CAT), ten international experts in antireflux surgery engaged in a structured three-round virtual Delphi consensus process. This iterative method aimed to identify mandatory, optional, and prohibited steps in the MSA procedure and to define key technical errors.

The resulting MSA CAT was supplemented with an operative manual detailing standardized steps and quality scoring criteria ranging from scores 1 (poor technique) to 4 (excellent technique). The tool was then tested for reliability by six surgeons who independently assessed two recorded MSA procedures. To evaluate interassessor reliability, assessments were compared across different reviewers. After three months, the same surgeons re-assessed the videos to evaluate intra-assessor (test-retest) reliability. Cohen’s kappa coefficient was used to quantify agreement levels.

Key Findings

The Delphi process refined the tool’s structure, ensuring consensus on critical operative steps and errors. The finalized MSA CAT included a detailed checklist of mandatory, optional, and prohibited surgical maneuvers, alongside a gradation of technique quality scores.

Reliability analysis demonstrated that the MSA CAT exhibited good consistency both between different assessors and for repeated assessments by the same surgeon. Mean interassessor Cohen’s kappa values were high at 0.785 (range 0.582-0.892) and 0.827 (range 0.591-0.912) for two separate videos, indicating substantial to almost perfect agreement. Intra-assessor reliability was similarly robust with mean kappa values of 0.745 (0.717-0.773) and 0.740 (0.510-1.00), reflecting good reproducibility over time.

Importantly, classification of scores into dichotomous categories—“concerning” versus “acceptable”—yielded the most consistent reliability, underscoring the tool’s clinical utility in identifying surgeons or cases requiring quality interventions.

Expert Commentary

The creation of the MSA CAT represents a significant advance in surgical quality assurance for antireflux procedures. As the first competency tool developed specifically for MSA and validated through an international expert consensus and objective reliability testing, it sets a precedent for rigorous intraoperative performance evaluation in prospective trials. Video-based assessment circumvents biases inherent in traditional credentialing or subjective appraisal and encourages transparency in surgical performance.

Limitations include the relatively small number of videos used for reliability testing and lack of outcome correlation at this stage. Future prospective validations, including retrospective video analyses planned within the GOLF trial, will be essential to confirm predictive validity and generalizability. The challenge remains to implement such tools widely across training programs and clinical practice to drive quality improvement beyond trial settings.

Conclusion

The MSA CAT is a pioneering, rigorously developed tool that reliably assesses competency in magnetic sphincter augmentation for GERD. Embedded within the GOLF trial’s broader surgical quality assurance program, it promises to enhance intraoperative quality monitoring, ultimately improving surgical outcomes. This methodology may serve as a model for competency assessment development in other emerging minimally invasive surgical techniques.

Funding and Clinical Trials Registration

The GOLF trial, including the development of the MSA CAT, is supported by international research collaborations. The trial is registered under ISRCTN: 1353317. Specific funding sources were not detailed in the source publication.

References

• Menon N, Goldsmith P, Owen R, et al. Development and Reliability of a Competency Assessment Tool for Magnetic Sphincter Augmentation of the Esophagus (MSA CAT) as Part of a Video-based Surgical Quality Assurance Program for the GOLF Trial. Ann Surg. 2026 Aug 6. PMID: 42557598.
• Shaheen NJ, Hansen RA, Morgan DR, et al. Acid suppression medications and risk of esophageal adenocarcinoma: a population-based nested case-control study. Gastroenterology. 2016;150(3): 605-615.
• Richter JE, Rubenstein JH. Presentation and epidemiology of gastroesophageal reflux disease. Gastroenterology. 2018;154(2):267-276.
• Swanstrom LL, Hansen PA. Minimally invasive magnetic sphincter augmentation for gastroesophageal reflux disease. Surg Clin North Am. 2018;98(2):327-337.