Objective Structured Assessment of Technical Skill in assessing technical competence to carry out caesarean section with increasing seniority

Since the incorporation of workplace-based assessment within the specialty training programme in obstetrics and gynaecology, the assessment of technical competence to carry out caesarean section has been undertaken by the Objective Structured Assessment of Technical Skill tool. This requirement has been formalised in the Matrix of Educational Progression, ensuring that the tool must assess trainees’ technical competence in caesarean section procedures of varying levels of complexity throughout training. Trainee feedback suggests that the effectiveness of the tool diminishes as the seniority of the trainee increases, with technical competence assessed less effectively in more complex procedures. This seems to be a result of the generic design of the tool and insufficient training on the part of assessors. Both of these are due to be addressed within a division of the Objective Structured Assessment of Technical Skill tool into explicitly formative and summative assessments of technical skill, following a General Medical Council-led consultation on the future of workplace-based assessment.

Assessing competencies within the specialty training programme

The introduction of a new specialty training programme (STP) in August 2007 heralded a new approach to the definition and assessment of competencies within obstetrics and gynaecology in the UK. In accordance with the principles of the National Health Service’s Modernising Medical Careers and the statutory regulator — then the Postgraduate Medical Education and Training Board, now the General Medical Council – the STP is underpinned by a curriculum, designed to ensure the achievement of standards agreed nationally.

The design of the curriculum ensures that the STP is competency based, rather than time-based, with educational progression assessed by the attainment of competencies stipulated within the curriculum rather than number of training hours completed. This assessment of educational progress is undertaken on an annual basis. Competencies are benchmarked and referenced against average timescales within a programme to measure appropriate progress at specific waypoints. These waypoints are created by the organisation of the 7-year STP into three distinct stages: basic, intermediate and advanced training.

The move from timed-based to competency-based training has constituted a significant challenge, one that has acted as a driver for cultural change in training within obstetrics and gynaecology. Nevertheless, this development was a necessity, not least as a result of the Working Time Regulations, applicable to junior doctors from August 2004. This ensured that, as the number of hours over the course of specialty training decreased, the traditional training model based on a long apprenticeship was no longer applicable. Furthermore, high-profile inquiries into misconduct and public expectation have ensured that doctors are required to demonstrate competency and also document evidence relating to the ongoing attainment of this in order to maintain public confidence in the profession. By identifying competencies (job-related tasks that use integrated elements of knowledge, skills and attributes), it is possible to ensure that health professionals carry out their clinical role to an appropriate standard, and make appropriate progress through a structured programme of specialty training.

At the heart of the move to competency-based training is the identification of important competencies that indicate either the attainment or ongoing demonstration of key knowledge, skills and attributes, and also the construction of effective ways to assess these. To this end, Postgraduate Medical Education and Training Board advised Royal Colleges to incorporate rigorous assessment strategies within their respective STPs, strategies that were required to reflect the overall objectives of the STP and provide documentary evidence of incremental competency attainment. As a consequence, the potential for workplace-based assessment (WPBA) was present throughout the new curriculum of the STP in obstetrics and gynaecology, to embed uniform and structured assessments for learning and to encourage a culture of documenting the achievement of clinical, educational and professional competencies.

Two types of workplace-based assessment can be identified: those in which a judgement of performance is based directly on observation of that performance; and those in which an aggregation of information is obtained from multiple sources over time. The former (direct performance measures) confine assessment to an individual encounter; as such, the assessment is time-bound and authentic, with performance assessed by a trainer using a generic rating form that reflects multiple competencies and, importantly, followed by one-to-one feedback. Examples include Objective Structured Assessment of Technical Skills (OSATS), Procedure-Based Assessment and Direct Observation of Procedural Skills. These are distinct from methods in which performance is assessed over a longer period of time, with judgements based on a broader exposure to the trainee’s work, often from multiple sources.

Workplace-based assessment offers a dimension of assessment different to that provided by traditional examinations. This is most commonly illustrated with reference to Miller’s pyramid of clinical competence assessment, which defines different assessment constructs, in terms of the nature of learning required and the situational context of this learning. The bottom two levels of the pyramid relate to the demonstration of knowledge and the application of this knowledge, respectively (‘knows’ and ‘knows how’), domains that have traditionally been tested in written examinations. The top two levels concern the assessment of practical demonstration and practical application of knowledge and skills, respectively (‘shows how’ and ‘does’). Although the ‘shows how’ domain can still encompass the practical elements of formal examinations, such as objective structured clinical examinations, the ‘does’ domain relates to assessment of habitual performance; that is, captured in the workplace. Such assessment seeks to assess performance rather than competence — what is done in actual professional practice rather than what is done in a controlled representation of this (i.e. the ‘shows how’ assessment of the objective structured clinical examinations, or of simulation in general).

An important feature of a successfully designed STP is that the various different assessments within it contribute effectively towards its objectives. One manifestation of this is the requirement for the traditional assessment elements, such as formal examinations on the one hand, and WPBA on the other, each to be contextualised sufficiently to inform decisions relating to a trainee’s progress within the STP. Furthermore, it is critical that assessments contribute towards such decisions with consistency (in the interests of fairness to trainees) and transparency (in the interests of patient reassurance). To this end, the Royal College of Obstetricians and Gynaecologists (RCOG) has developed a Matrix of Educational Progression, which documents the requirements for annual progression and, importantly, for progression at the waypoints of the STP. It is at these waypoints that the contribution of both formal examination and WPBA is most explicitly evident: in each case, in order to progress to the next level of training, the trainee is required, among other things, to have attained success both in the appropriate examination (either Part 1 MRCOG or Part 2 MRCOG) and in the OSATS assessments stipulated in the Matrix. The inclusion of such divergent assessment criteria for progression within the STP is evidence of the detailed design of its curriculum and the usefulness of frameworks such as Miller’s pyramid in such endeavours.

Objective Structured Assessment of Technical Skill in obstetrics and gynaecology

Within a curriculum of the scope and detail of that of the STP for obstetrics and gynaecology, the competencies identified will vary in nature: some will be technical procedures, assessed through direct observation; others will be non-technical skills or attributes, assessed through longer-term multisource feedback. Equally, not all competencies will have the same importance attached to them. All will contribute to the overall objectives of the STP, but some will be more significant in contributing towards decisions regarding a trainee’s progression. It is this significance attached to certain technical procedures that motivated the RCOG to introduce the OSATS tool into the curriculum of the new STP. A small number of procedures were deemed sufficiently fundamental to the practice of obstetrics and gynaecology that an objective assessment tool was required to contribute towards the assessment process. In 2005, an RCOG working party identified six core competencies in which to pilot a method of assessing the technical skills they entailed: caesarean section; operative vaginal delivery; perineal repair; manual removal of the placenta; evacuation of the uterus; and diagnostic laparoscopy.

The OSATS tool that the RCOG used in the pilot was developed in the USA in the 1990s. The original tool consisted of a technical checklist specific to the procedure, as well as 10 global items common to all procedures each rated 0–4. This list was modified to reduce the number of global items to seven and replaced the global scoring scale with a behaviourally based spectrum using descriptors. The OSATS tool introduced by the RCOG adapted this version, keeping both the checklist and seven global elements ( Fig. 1 ), the latter being amended by the combination of existing items and the introduction of new ones such as ‘Documentation of Procedures’. The inclusion of the technical checklist distinguished the OSATS from similar tools such Direct Observation of Procedural Skills, which use a more generic global scoring approach alone. After a successful pilot, OSATS was formally introduced for all levels of trainee as part of the newly launched STP for obstetrics and gynaecology in August 2007 ( Fig. 1 ).

Fig. 1
Objective Structured Assessment of Technical Skills form for caesarean section. © Royal College of Obstetricians and Gynaecologists; reproduced with permission.

The attraction of using OSATS was clear: the tool provided the prospect of reproducible assessment, which impinges minimally on clinical efficiency during operating lists. Bringing the tool into the operating theatre and away from the laboratory, where it was initially deployed using bench models, could avoid the time and cost disadvantages associated with its early use, while marrying up technical and decision-making skills. Early research indicated good acceptability levels, with both trainers and trainees valuing the capacity of the OSATS tool to assess surgical skills in a ‘real’ setting. The limited benefit of assessing purely technical skills in certain situations, however, was identified even before the formal introduction of the tool; for example, it is much more difficult to decide whether or not to carry out an emergency caesarean section than it is to carry out the procedure itself. The assessment of these related, non-technical, skills constitutes a challenge that the RCOG is still seeking to address in the most effective way; to this end, a labour-ward pilot of the Non-Technical Skills for Surgeons tool is planned to start in August 2012.

The number of competencies assessed by OSATS within the STP for obstetrics and gynaecology has increased from six to 10, with addition of the following procedures: fetal blood sampling; diagnostic hysteroscopy; opening and closing the abdomen; and operative laparoscopy. For any of the 10 competencies to be signed off in a trainee’s logbook, each OSATS must have been completed successfully on a minimum of three occasions and assessed by two different trainers, one of whom should be a consultant. An OSATS is completed successfully when all the items on the technical checklist are ticked as ‘performed independently’, and most of the global items are ringed either the middle or the right of the scale, with ‘insight/attitude’ consistently judged as ‘fully understands areas of weakness’. The initial ‘pass’ and ‘fail’ judgements have been replaced by ‘competent in all areas of this OSATS’ and ‘working towards competence’, emphasising the fact that a trainee is not expected to complete successfully an OSATS at the first attempt. Once signed up for ‘independent practice’ on a given procedure, it is recommended that an annual OSATS is undertaken by the trainee to demonstrate continued competency in that area.

Objective Structured Assessment of Technical Skill in obstetrics and gynaecology

Within a curriculum of the scope and detail of that of the STP for obstetrics and gynaecology, the competencies identified will vary in nature: some will be technical procedures, assessed through direct observation; others will be non-technical skills or attributes, assessed through longer-term multisource feedback. Equally, not all competencies will have the same importance attached to them. All will contribute to the overall objectives of the STP, but some will be more significant in contributing towards decisions regarding a trainee’s progression. It is this significance attached to certain technical procedures that motivated the RCOG to introduce the OSATS tool into the curriculum of the new STP. A small number of procedures were deemed sufficiently fundamental to the practice of obstetrics and gynaecology that an objective assessment tool was required to contribute towards the assessment process. In 2005, an RCOG working party identified six core competencies in which to pilot a method of assessing the technical skills they entailed: caesarean section; operative vaginal delivery; perineal repair; manual removal of the placenta; evacuation of the uterus; and diagnostic laparoscopy.

The OSATS tool that the RCOG used in the pilot was developed in the USA in the 1990s. The original tool consisted of a technical checklist specific to the procedure, as well as 10 global items common to all procedures each rated 0–4. This list was modified to reduce the number of global items to seven and replaced the global scoring scale with a behaviourally based spectrum using descriptors. The OSATS tool introduced by the RCOG adapted this version, keeping both the checklist and seven global elements ( Fig. 1 ), the latter being amended by the combination of existing items and the introduction of new ones such as ‘Documentation of Procedures’. The inclusion of the technical checklist distinguished the OSATS from similar tools such Direct Observation of Procedural Skills, which use a more generic global scoring approach alone. After a successful pilot, OSATS was formally introduced for all levels of trainee as part of the newly launched STP for obstetrics and gynaecology in August 2007 ( Fig. 1 ).

Fig. 1
Objective Structured Assessment of Technical Skills form for caesarean section. © Royal College of Obstetricians and Gynaecologists; reproduced with permission.

The attraction of using OSATS was clear: the tool provided the prospect of reproducible assessment, which impinges minimally on clinical efficiency during operating lists. Bringing the tool into the operating theatre and away from the laboratory, where it was initially deployed using bench models, could avoid the time and cost disadvantages associated with its early use, while marrying up technical and decision-making skills. Early research indicated good acceptability levels, with both trainers and trainees valuing the capacity of the OSATS tool to assess surgical skills in a ‘real’ setting. The limited benefit of assessing purely technical skills in certain situations, however, was identified even before the formal introduction of the tool; for example, it is much more difficult to decide whether or not to carry out an emergency caesarean section than it is to carry out the procedure itself. The assessment of these related, non-technical, skills constitutes a challenge that the RCOG is still seeking to address in the most effective way; to this end, a labour-ward pilot of the Non-Technical Skills for Surgeons tool is planned to start in August 2012.

The number of competencies assessed by OSATS within the STP for obstetrics and gynaecology has increased from six to 10, with addition of the following procedures: fetal blood sampling; diagnostic hysteroscopy; opening and closing the abdomen; and operative laparoscopy. For any of the 10 competencies to be signed off in a trainee’s logbook, each OSATS must have been completed successfully on a minimum of three occasions and assessed by two different trainers, one of whom should be a consultant. An OSATS is completed successfully when all the items on the technical checklist are ticked as ‘performed independently’, and most of the global items are ringed either the middle or the right of the scale, with ‘insight/attitude’ consistently judged as ‘fully understands areas of weakness’. The initial ‘pass’ and ‘fail’ judgements have been replaced by ‘competent in all areas of this OSATS’ and ‘working towards competence’, emphasising the fact that a trainee is not expected to complete successfully an OSATS at the first attempt. Once signed up for ‘independent practice’ on a given procedure, it is recommended that an annual OSATS is undertaken by the trainee to demonstrate continued competency in that area.

Objective Structured Assessment of Technical Skill and caesarean section

Since the introduction of OSATS to assess lower-segment caesarean section (LSCS) in August 2007, over 40,000 individual encounters have been undertaken as part of the STP in obstetrics and gynaecology. After the creation of the Matrix of Educational Progression in 2011, however, the rationale of each encounter has become more defined. In outlining the requirements for trainee progression with regard to OSATS, and indeed other WPBA tools, the Matrix identifies three distinct purposes for OSATS: the demonstration of evidence of training in practical skills; confirmation of competence; and confirmation of continued competence. In the case of LSCS, the demonstration of evidence of training in practical skills takes place during the first year of training (ST1). The confirmation of continued competence, meanwhile, can take place from ST3 to ST7, depending on the complexity of the case. It is, then, the OSATS undertaken for the purpose of confirming competence at LSCS that is pertinent to progression through the waypoints of the STP. Once the requirement to confirm competence at opening and closing the abdomen is achieved by the end of ST1, three further OSATS are needed to confirm competence at LSCS during the course of the STP (with trainees first having demonstrated competence at the remaining individual stages of the procedure, from incising the visceral peritoneum to careful checking of the ovaries and tubes). One of these OSATS is located at the final year of each stage of training, with successful completion a criterion for exit from that stage of training.

For the OSATS to assess competence effectively at LSCS in the three stages of training, three different levels of complexity are identified for the procedure: uncomplicated; intermediate; and complex. The identification of these categories ensures that, for the procedure, an OSATS is undertaken to reflect the level of complexity expected at the relevant stage of training. The distinction, however, becomes more pertinent in the case of those OSATS encounters undertaken to confirm competence, given that successful completion is a requirement of progression within the STP. In order to exit basic training (ST2), a trainee is required to confirm competence in an uncomplicated LSCS, such as a straightforward category 4 caesarean section or a LSCS on a woman with not more than one previous section. To advance from intermediate training (ST5), meanwhile, a trainee is required to confirm competence in an intermediate LSCS, such as in the case of a transverse fetal lie. Finally, to attain a certificate of completed training at the end of advanced training (ST7), a trainee is required to confirm competence in a complex LSCS, for example with the presence of fibroids in the lower uterine segment. Related complex competencies, such as LSCS in the context of major placenta praevia or maternal BMI above 35, are requirements for the Advanced Labour Ward Practice Advanced Training Skills Module, the completion of which is a prerequisite for many consultant posts but not mandatory to attain a certificate of completed training.

With the different levels of complexity defined, there is potential for the LSCS OSATS to retain its relevance, as an assessment that informs progression while providing a learning benefit, throughout training. To gain some understanding of the extent to which this potential is realised, it is instructive to look at trainees’ experiences of LSCS assessment using the OSATS tool at the three levels of training:

‘I am currently nearing the end of my ST1 year. I think the OSATS form itself is user friendly and provides a logical, clear and concise picture of how the assessor thinks we performed. It goes through a caesarean section in a stepwise manner, asking if the assessor feels we performed each step independently or need further help. This allows us to look for areas of weakness during the process and not just get an overall picture. The ‘generic technical skills’ descriptions for each section are quite ‘wordy’, but if the assessor takes the time to read them and fill them in, this really helps the trainee to gauge how technically skilled the assessor thinks we are. It can’t be a yes/no answer system, as technical skills are acquired, practised and adapted until competence, so we can see our progression throughout the year. This, however, can be a drawback if the assessor does not take time to read this section properly, as it does require more attention than simple yes/no answers. Overall I feel that the caesarean section OSATS, if done properly, is a good assessment tool and feel I have learnt a lot about my progress throughout this year by looking at my completed forms.’ Basic Level Trainee (ST1)
‘At an intermediate level, the OSATS format remains identical to those used at a basic level and therefore is less useful as a training tool and difficult to use for assessments. For example, when assessing a caesarean section for placenta praevia, there is no way of remarking on the specific steps that you took in order to perform the high-risk procedure safely – or what you did to overcome any problems. The comments box is also too small to write everything in. Essentially the form is too broad. I think that OSATS could be successfully implemented at an intermediate level if there were more specific steps. For example there could be boxes for skills such as ‘Took appropriate steps to control surgical bleeding’, ‘administered suitable medical treatment for atonic uterus’, and ‘divided through layers of adhesions carefully’. Intermediate Level Trainee (ST4)
‘Although I found the caesarean section OSATS useful as a basic level trainee, especially when moving from ST2 to ST3 in demonstrating competence with the procedure, I have found the current format doesn’t provide much relevance to the types of delivery I am performing halfway through advanced training, and especially the ATSM in Advanced Labour Ward Practice. For example, deliveries in women with multiple fibroids, or placenta praevia require techniques and manouevres which assessors have no place to comment on except in the small free-text box. So it may be that a more generic OSATS is required with more flexibility for commenting on these more complex procedures. I also feel that space for the trainee to reflect upon the procedure and assessment would be valuable. If there is something that has gone particularly well, or badly, during an assessment, I currently jot it down at the bottom of the form.’ Advanced Level Trainee (ST6)

In spite of the sound logic of identifying different levels of complexity of the LSCS procedure to enable the effective assessment of competence in this fundamentally important area of obstetrics and gynaecology over 7 years of run-through training, the vignettes seem to suggest that perceptions of the assessment’s utility in its current format are not uniform at different levels of trainee seniority. Rather, it is possible that the OSATS tool for assessing LSCS becomes less useful to a trainee as they advance through training. The tool would seem to serve its purpose effectively in basic training, through the provision of the formative function of framing the steps for further development and nurturing confidence, and the summative function of demonstrating the competence required to progress to the next stage of training. At both intermediate and advanced levels, however, there is a sense that the tool loses its ability to capture effectively the more complicated elements that require demonstration in these latter two stages of training.

Only gold members can continue reading. Log In or Register to continue

Stay updated, free articles. Join our Telegram channel

Nov 8, 2017 | Posted by in OBSTETRICS | Comments Off on Objective Structured Assessment of Technical Skill in assessing technical competence to carry out caesarean section with increasing seniority

Full access? Get Clinical Tree

Get Clinical Tree app for offline access