Objective
To develop and validate an instrument to assess surgical skills during vaginal surgery.
Study Design
Trainees from 2 institutions were directly assessed in the operating room by supervising surgeons while performing a vaginal hysterectomy using the new Vaginal Surgical Skills Index, global rating scale, and visual analogue scale. Trainees were assessed again by the same surgeons 4 weeks after the live surgery and by a blinded outside reviewer using a videotape of the case. Internal consistency, interrater and intrarater reliability, and construct validity were evaluated.
Results
Two hundred twelve evaluations were analyzed on 76 surgeries from 27 trainees. There was good internal consistency, interrater, and intrarater reliability. Vaginal Surgical Skills Index scores correlated with global rating score and visual analog scale scores. Increasing Vaginal Surgical Skills Index scores significantly correlated with year of training and surgical volume with an estimated increase in score of 0.3 per hysterectomy performed.
Conclusion
The Vaginal Surgical Skills Index is a feasible, reliable, and valid instrument to assess vaginal surgical skills.
Despite advances in surgical instruments and techniques, progress in formalized assessment of surgical skills for obstetrics and gynecology residents is limited. A 1997-1998 survey of obstetrics and gynecology residency directors in the United States found that only 29% of programs had any formal surgical curricula and only 17% used any method of standardized assessment of resident technical skills. Since this survey, many residency programs have improved their assessment strategies; however, assessments of surgical skills continue to be limited to nonvalidated global assessments by supervising surgeons at the end of a surgical rotation and by the use of surgical case logs. Case logs lack content validity because they only document participation in a case and do not assess the technical ability of the individual performing the procedure and most forms of global assessment have poor reliability and unknown validity. Although surgical checklists are also useful for learning and giving feedback, for assessment purposes, global rating scales scored by experts have higher reliability and validity than checklists. However, there are currently no published assessment instruments that have been tested for reliability or validity for use in live open gynecologic or vaginal surgery.
Formal intraoperative assessment is difficult to standardize because the difficulty level of the procedure and the amount of participation from each trainee during the case is variable. Cadaveric specimens, live animals, and bench top simulators have been used in an attempt to standardize the assessment of basic surgical skills outside the operating room. Objective Structured Assessments of Technical Skills in these settings have good reliability and validity using animal models and bench top simulators, yet they are limited by their availability, cost, and need for faculty time. Moreover, facilities required to support these programs may not be feasible for smaller community training programs. Therefore, the gold standard of determining a surgeon’s competence remains assessment during live surgery and valid and reliable intraoperative assessment tools should be developed and validated in gynecology for this purpose. These tools could also be beneficial in providing accurate documentation of a trainee’s performance through time allowing for directed teaching and could be used to provide a template for providing constructive formative feedback about their performance.
Intraoperative assessment instruments should be psychometrically robust, feasible to administer, and easily adaptable by residency programs. The most commonly used instrument to evaluate surgical skills is the global rating of operative performance developed by Dr Reznick and his general surgery colleagues. This instrument consists of a 7-item global rating scale that allows supervising surgeons to rate important but generic skills during surgical procedures including: respect for tissue, time and motion, instrument handling, knowledge of instruments, use of assistants, flow of operation, and knowledge of the procedure. Although this performance index has been shown to be reliable and valid when used with bench top simulators and animal models, its psychometric properties have not been demonstrated using blinded reviewers in live surgery, despite it being the only tool listed on the Accreditation Council for Graduate Medical Education Patient Care Assessment section for the assessment of surgical skills. Furthermore, although, this instrument has been used in the gynecologic literature, albeit sparingly, its psychometric properties have not been determined for live surgery and there are currently no reliable and valid instruments for assessing vaginal surgical skills in the laboratory or in the operating room setting.
The primary aims of this study were to develop a feasible, reliable, and valid instrument to evaluate surgical skills during vaginal surgery and to compare the psychometric properties of this new assessment scale for vaginal surgery with the more commonly used Global Rating Scale (GRS) of Operative Performance. Our goal was to develop an assessment tool that is representative of essential skills required to perform vaginal surgery. We hypothesized that the Vaginal Surgical Skills Index (VSSI) would be a feasible, reliable, and valid instrument for the assessment of vaginal surgical skills while performing vaginal hysterectomy.
Materials and Methods
To develop an instrument specifically for the evaluation of vaginal surgical skills, we expanded and modified the original 7-item GRS. Additional items were included such as initial inspection, performance of an incision, maintenance of visibility, use of electrosurgery, knot tying, maintenance of hemostasis, removing fluid and debris at the completion of the procedure, and forward planning for a total of 13 items ( Appendix ). These additional items were deemed important and highly relevant during vaginal surgery based on feedback from practicing gynecologic surgeons given the unique aspects of operating using a vaginal approach. The descriptors of the GRS were also rewritten to provide clear descriptions for the assessor of what was to be rated and these descriptors were added to all items rather than every other item on the GRS. To rate the trainee’s overall performance during the procedure, a 100-mm visual analog scale (VAS) was also included as an additional measure of the attending’s global impression of surgical skill by the trainee. Content and face validity of the VSSI were assessed by surveying gynecology trainees and a panel of teaching gynecologic surgeons from 3 training institutions as to whether the scale appeared fair and appropriate in assessing important qualities of surgical technique during vaginal surgery. Because this project was presented to, discussed at, and funded by the Society of Gynecologic Surgeons Education Research Forum in 2006, we also benefited from input from a wider group of gynecologic surgeons from multiple training institutions.
Institutional Review Board exemption was obtained as this project involved the use of educational tests and did not affect the clinical course of the patients. We evaluated obstetrics and gynecology residents and postresidency, Female Pelvic Medicine and Reconstructive Surgery fellows during the performance of a vaginal hysterectomy at the Cleveland Clinic and Mayo Clinic from May 2007 to June 2008 using the new VSSI, GRS, and VAS. Trainees were evaluated several times over the course of their gynecology rotation. Trainees were instructed to perform the hysterectomy as if they were the primary surgeon and neither the assistant nor the supervising surgeon would interfere with decisions as long as the patient’s safety was maintained. The vaginal hysterectomy was videotaped from the incision to completion with removal of the uterus and inspection of all pedicles for hemostasis while preserving the anonymity of the trainee. As the vaginal hysterectomy was often performed with other procedures such as salpingo-oophorectomy or prolapse surgeries, we did not include vaginal cuff closure in the video recording. Because the video recording did not include audio, the supervising attending and other surgical assistants were instructed not to assist or correct the trainee verbally; instead, every effort was made to correct or guide the trainee with hand gestures so that it was clear on the video recording that the trainee had made an error and was being rectified or the trainee did not know the steps of the procedure. Assistants were also instructed not to automatically assist but to have the trainee demonstrate what assistance was needed because the use of assistance was 1 of the evaluation parameters on the VSSI form. The VSSI, GRS, and VAS were completed by a supervising attending surgeon immediately after observing the trainee perform a vaginal hysterectomy. Additional information collected included the time taken to perform the hysterectomy and to fill out the VSSI form, training level of the trainee, and number of vaginal hysterectomies the trainee had previously performed. The attending surgeon also rated the difficulty of the case by answering the question, “Compared with average level of difficulty for your operations, would you say this operation was?” and checking a 5-point discrete response scale ranging from “much easier than my average operation” to “much more difficult.“
To evaluate interrater reliability, a copy of the blinded video-recorded procedure was sent to a surgeon at a separate institution who evaluated the trainee’s performance using the same assessment forms after watching the procedure. Another copy of the recorded procedure was given to the intraoperative supervising physician for review and form completion at least 4 weeks after the date of the original procedure to assess intrarater reliability. Both evaluations were performed in a blinded manner with the anonymity of the trainee and their training level preserved as the recordings only included the date of surgery and the trainee 4-digit unique identifier.
Cronbach’s alpha was calculated to assess internal consistency of the VSSI, assuming independence across surgeries. Interrater reliability and intrarater reliability were analyzed using Intraclass correlation coefficient (ICC). Construct validity was evaluated by measuring convergent validity, or the association between VSSI scores (range, 0–52) and the GRS scores (range, 0–35) or VAS scores (range, 0–100) using Pearson correlation coefficient (r) and by discriminant validity by comparing VSSI scores with training level and surgical volume. Because most trainees had multiple surgeries assessed, the estimated mean and standard error of the instrument scores by training level were compared. Based on previous studies validating the GRS, at least 20 trainees would be necessary to discriminate between training levels to demonstrate discriminate validity. Linear mixed effect models were used to evaluate the relationship between the VSSI scores and training level or surgical volume while considering the random trainee effect and the correlation within surgery. All analyses were performed using SAS 9.1 software (SAS Institute, Inc, Cary, NC).
Results
A total of 27 trainees postgraduate training levels (PGY) 1-7 (residents and fellows) from 2 institutions participated in this study. Because most trainees had multiple procedures assessed during the study period, there were a total of 76 surgeries performed. Five supervising surgeons from 2 institutions (C.C. and M.C.) completed evaluations immediately after each procedure and then again after watching the recorded procedure. One surgeon from a third institution (University of California, San Francisco) served as the blinded external reviewer who watched all the recorded procedures and completed the evaluations for each of the procedures. A total of 212 surgical evaluations were analyzed. The mean (± standard deviation [SD]) time taken to complete vaginal hysterectomy was 36.6 ± 17.8 minutes (range, 8–135) and the mean time taken to complete VSSI form was 2 ± 1 minutes (range, 1–10).
Internal consistency for the VSSI and GRS were high (Cronbach’s alpha = 0.95–0.97). As shown in Table 1 , interrater reliability (ICC = 0.53) and intrarater reliability (ICC = 0.82) for the VSSI were also good; however, the GRS had the lowest interrater reliability at 0.31 and intrarater reliability was 0.64 ( Table 2 ).
| Evaluation method | VSSI a | GRS a |
|---|---|---|
| Live surgery | 0.96 | 0.95 |
| Blinded videotape review by the same surgeon after 1 mo | 0.97 | 0.94 |
| Blinded external videotape review | 0.95 | 0.94 |
| Assessment scale | Interrater reliability a | Intrarater reliability a |
|---|---|---|
| VSSI | 0.53 | 0.82 |
| GRS | 0.31 | 0.64 |
| VAS | 0.51 | 0.69 |
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
