Highlights
In a prospective observational study of 32 otolaryngology residents, mastoidectomy performance on 3D-printed temporal bones improved with advancing postgraduate year and with longer time in training.
A validated 14-item final product analysis checklist showed excellent inter-rater reliability, with an intraclass correlation coefficient of 0.9210, supporting consistent scoring across expert faculty.
Senior residents outperformed junior residents, and a proficiency threshold of 45 out of 70 was established using the contrasting groups method, suggesting a practical benchmark for competency-based education.
The study strengthens the case for combining standardized simulation models with objective end-product scoring for both formative feedback and potentially higher-stakes summative assessment.
Background
Mastoidectomy is a foundational but technically demanding procedure in otologic surgery. It requires precise drilling within a compact anatomic space that contains critical structures including the facial nerve, sigmoid sinus, labyrinth, ossicles, and dura. Traditional mastery depends on graduated operative exposure, faculty supervision, and temporal bone dissection. However, modern residency training faces several constraints: reduced operative autonomy, work-hour limits, variable case volumes, and restricted access to cadaveric temporal bones. These pressures have accelerated interest in simulation-based training and more rigorous methods of skill assessment.
Three-dimensional printed temporal bone models are particularly attractive because they are reproducible, scalable, and safer and more accessible than cadaveric specimens. Yet simulation is only as useful as the assessment framework that accompanies it. Surgical educators increasingly seek tools that are objective, reliable, and linked to competency progression rather than case counts alone. In mastoidectomy, this means not only observing how a learner drills, but also evaluating the quality of the completed dissection against predefined technical standards.
The study by Helou and colleagues addresses this need by testing whether a standardized 3D-printed temporal bone model, assessed by final product analysis, can discriminate performance across residency levels and provide a defensible objective structured assessment of technical skills.
Study Design
Design and setting
This was a prospective observational study conducted at a single academic otolaryngology residency program over three years.
Participants and procedures
Thirty-two residents performed a total of 64 mastoidectomies on 3D-printed temporal bone models. The abstract indicates repeated assessments over time, allowing evaluation of both cross-sectional differences by postgraduate year and longitudinal trends across attempts.
Assessment method
Three expert faculty members independently and anonymously scored each completed mastoidectomy using a validated 14-item final product analysis checklist. Final product analysis focuses on the quality of the completed surgical field rather than solely on the process of instrument handling. In this context, that likely includes adequacy of exposure, preservation of key landmarks, and avoidance of technical errors affecting critical structures. The total possible score was 70.
Statistical analysis
The investigators used linear regression to test associations between performance and training level or duration in residency. Group comparisons were performed with ANOVA, Mann-Whitney U, and Wilcoxon signed-rank tests as appropriate. Inter-rater reliability was measured using the intraclass correlation coefficient. A proficiency cutoff was derived using the contrasting groups method, a standard psychometric approach that identifies a score separating more and less experienced performers.
Key Findings
Performance increased with training level
Median mastoidectomy scores rose steadily from 35.0 in postgraduate year 1 to 53.7 in postgraduate year 5, out of a maximum 70. This gradient is clinically important because it suggests the tool has construct validity: residents with greater experience performed better, as would be expected if the assessment is truly measuring technical skill rather than noise or rater preference.
Linear regression showed a significant positive association between performance and both postgraduate year and time in residency. That finding supports the idea that the scoring system tracks progressive skill acquisition rather than serving as a one-time snapshot.
Senior residents clearly outperformed junior residents
When residents were grouped by experience, senior trainees in PGY 4 and PGY 5 achieved a median score of 54.0, compared with 38.0 for junior trainees in PGY 1 through PGY 3. This difference was statistically significant at p = 0.012. In practical terms, the gap is substantial and suggests the assessment may be useful for distinguishing learners who are approaching independent competence from those still building foundational drilling skills.
Scores improved over repeated attempts
Across all training levels, median scores increased from 43.2 on the first attempt to 52.5 on the third attempt. Although this trend did not reach statistical significance, it remains educationally relevant. The lack of significance may reflect limited sample size, heterogeneity across training levels, or insufficient power to detect within-subject change. The direction of effect is consistent with learning through deliberate practice.
Inter-rater reliability was excellent
The intraclass correlation coefficient was 0.9210 with p < 0.0001, indicating excellent agreement among the three expert scorers. This is one of the strongest findings in the study. Reliable scoring is essential if simulation-based assessments are to be used beyond informal feedback. An ICC above 0.9 suggests that the checklist and the standardized 3D-printed models together produce a highly reproducible assessment environment.
A proficiency benchmark was defined
Using the contrasting groups method, the investigators established a proficiency cutoff score of 45 out of 70. This is a notable translational step. Many simulation studies show that a tool can differentiate novices from experts, but fewer define a threshold that could be used to decide whether a trainee has met a minimum standard. While the optimal application of such a cutoff would require broader validation, having an empirically derived benchmark is important for competency-based curricula.
Why This Matters for Clinical Training
The study addresses a persistent challenge in surgical education: how to assess competence fairly when operative opportunities vary and subjective impressions can dominate faculty evaluations. Mastoid surgery is especially well suited to simulation-based assessment because the anatomy is intricate, complication stakes are high, and repeated cadaveric practice is often limited by cost and access.
Several features make this approach attractive. First, 3D-printed temporal bones can standardize task difficulty. This reduces case-to-case variability that is unavoidable in live surgery and even in cadaveric specimens. Second, final product analysis captures whether the learner achieved the anatomic and technical goals of the procedure. Third, the strong inter-rater reliability suggests that trained evaluators can apply the checklist consistently, improving fairness across residents.
For program directors and competency committees, this type of tool could complement operative logs, direct observation, and milestone-based assessment. It may be particularly useful at transition points, such as determining readiness for more autonomous otologic cases or identifying residents who need targeted remediation.
Expert Commentary and Critical Appraisal
This study has several strengths. It was prospective, included multiple years of training, used anonymous independent expert scoring, and employed a validated checklist with excellent reliability. The establishment of a proficiency cutoff increases its practical value. These are meaningful advances in an area where many educational interventions remain descriptive rather than psychometrically grounded.
Still, several limitations should temper interpretation. The study was conducted at a single academic center, which may limit generalizability. Local teaching culture, prior simulation exposure, and resident case mix could influence performance. The total sample size was modest, and although 64 procedures provide useful data, the study may have been underpowered to detect improvement across repeated attempts. The abstract does not provide confidence intervals, item-level performance data, or details on the exact features of the 3D-printed model, all of which would help readers judge precision and external applicability.
Another key question is transfer validity: does a higher score on a printed temporal bone translate into better operating room performance, fewer technical errors, or earlier entrustment in live mastoid surgery? The current study supports construct validity and reliability, but not yet direct patient-level or intraoperative outcome validity. That said, in competency-based medical education, simulation-based metrics do not need to replace clinical assessment; they can strengthen it by adding standardized evidence of technical capability.
The choice of final product analysis also deserves consideration. End-product scoring is efficient and objective, but it does not fully capture intraoperative judgment, tissue handling, economy of motion, or response to unexpected findings. The most robust assessment strategy may therefore combine product-based scoring with process-based evaluation, such as global rating scales or motion analysis. Even so, for mastoidectomy, the quality of the final dissection is highly relevant and clinically meaningful.
Context Within the Existing Literature
Simulation and objective technical assessment in otolaryngology have been developing for more than a decade. Prior work has shown that temporal bone simulation can differentiate experience levels and support deliberate practice, but consistency of models and defensible assessment methods have remained barriers to widespread adoption. Cadaveric dissection retains high fidelity, yet it is expensive, logistically demanding, and difficult to standardize.
The present study fits well with the broader movement toward mastery learning and competency-based progression. In other surgical domains, benchmarked simulation tasks with predefined passing standards have helped structure training before independent operative participation. The main contribution here is bringing that logic into mastoidectomy with a reproducible physical model and a high-reliability scoring framework.
Implications for Practice and Policy
For residency programs, this work supports investment in structured temporal bone simulation curricula that include repeated practice, expert feedback, and formal scoring. A cutoff such as 45 out of 70 should not be used rigidly in isolation, but it could serve as one component of a multimodal assessment portfolio.
For faculty, the findings encourage greater use of explicit checklists and blinded product review, which may reduce halo effects and interpersonal bias. For institutions and specialty boards, the study raises the possibility that simulation-based technical assessment could eventually contribute to summative decisions, provided multicenter validation confirms fairness and real-world relevance.
There may also be equity advantages. Programs with limited cadaveric resources could use 3D-printed temporal bones to provide more uniform access to procedural rehearsal and assessment. This is especially important in rare or complex procedures where case volume alone is an imperfect surrogate for competence.
Conclusion
Helou and colleagues provide compelling early evidence that mastoidectomy assessment using 3D-printed temporal bones and final product analysis is feasible, objective, and highly reliable. Performance increased with training level, senior residents scored significantly higher than junior residents, and the scoring system achieved excellent inter-rater agreement. The derived proficiency threshold of 45 out of 70 adds practical value for competency-based education.
The study does not answer every question, particularly around generalizability and transfer to operating room outcomes, but it meaningfully advances the science of otologic skills assessment. For otolaryngology educators seeking fairer and more standardized ways to evaluate technical development, this approach appears promising for both formative feedback and carefully designed summative use.
Funding and ClinicalTrials.gov
Funding was not reported in the abstract provided. No ClinicalTrials.gov registration number was reported; formal registration is often not applicable to observational surgical education studies.
References
Helou V, Khalil L, Perez PL, McCall AA, Jabbour N. Objective Structured Assessment of Technical Skills in Mastoidectomy Using 3D-Printed Temporal Bones. The Laryngoscope. 2026-05-08. PMID: 42103557.
Francis HW, Masood H, Chaudhry KN, et al. Objective assessment of mastoidectomy skills in the operating room and laboratory setting. Otolaryngology–Head and Neck Surgery. This body of work helped establish structured assessment approaches in temporal bone surgery.
Butler NN, Wiet GJ. Surgical simulation for training in mastoidectomy and other temporal bone procedures: relevance to competency-based education. Otolaryngologic Clinics of North America. Review literature supports simulation as an adjunct to limited cadaveric and operative exposure.
McRackan TR, Abdellatif WM, Wanna GB, Rivas A, Gupta N, Dietrich MS, et al. Evaluation of efficacy of simulation-based temporal bone training. Studies in otology education have repeatedly shown discrimination between levels of experience and improvement with deliberate practice.
Ericsson KA. Deliberate practice and acquisition of expert performance: a general overview. Academic Emergency Medicine. Although not otology-specific, this framework underpins repeated simulation practice and structured feedback in procedural learning.
Readers should verify full bibliographic details for background references according to institutional database access, as only the index study citation was supplied in the source material.