Using mixed reality for immersive surgical training

Our clients, Dr Stefan Hudson, Clinical Innovation Fellow, and Mr Ryan Kerstein, Consultant Plastic Surgeon at the Buckinghamshire NHS Trust, had an idea that could transform burns training for surgeons and significantly reduce cost for the NHS. They reached out to Team Consulting expressing current challenges during burn assessment and care. Our client wanted to explore using a mixed reality concept, a blend of physical and virtual environments, to train their surgical staff. By harnessing the power of mixed reality, immersive and customisable burn simulations can be created quickly and cost-effectively.

Simulating a burn for surgical training

Simulating a burn for surgical training is an intricate process. A prosthetic artist has to hand paint sheets that are then overlaid onto a mannequin. Most mannequins cost about £120K each. In addition, all the mannequins are of similar skin tone and age. This means that surgeons are not exposed to a great variety of patients or burn scenarios. This lack of training is often associated with a lack of confidence when transitioning to a real life burn patient.

A cost-effective burn training model for surgeons

Team designed a concept training tool which allows surgeons to train in highly-customisable and realistic burn simulations by using a mixed reality headset that maps ‘custom digital burns’ onto a mannequin. This allows surgical staff to get exposure across a diverse range of patient characteristics and burn scenarios. It is crucial for training surgeons to gather feedback from their trainer on their assessment of a burn. Therefore, a live-stream feature was considered in the UX. This headset concept creates a richer experience, reduces cost and boosts confidence as surgeons refine their skills through optional assessment prompts.

De-risking the technology at minimal cost

Before undertaking any significant development of this mixed reality concept, we needed to prove that the technology was feasible. To de-risk the technology, our software engineering team combined existing software packages with phone apps and custom code to build a demonstrator in two days. This gave us confidence that the technology was de-risked and that we could move forward with refinement of the prototype and UX/UI activities.

Innovating through collaboration and contextual research

Our team of UX designers and software specialists collaborated closely with our client to identify opportunities for design improvements in their training model. The project team conducted contextual research by taking part in the NHS’ burn simulation course at Stoke Mandeville hospital. This allowed us to see first-hand the challenges training surgeons face when responding to different burn scenarios and assessing their patients.

Re-creating different types of burns using a mixed reality headset

We first worked on creating the ability to customise a burn based on a patient’s physical characteristics which can affect the appearance of the burn. These provide context of the scenario to the training surgeon to aid in their assessment. The user experience team then focused on creating parameters for the burn itself: depth, size, cause and location to give surgeons different patient scenarios. The user interface of the headset displays a tailored ‘digital burn’ on top of the mannequin, ready for the training surgeons to begin their assessment.

Software engineering: redefining what technology can achieve

We needed to ensure that the concept could be proved with minimal effort and investment so, as a first step, we focused on developing the simulator based on a mannequin arm. The technology could later be developed to cover an entire body.

The headset needed to detect and track the movement of the ‘real-world’ mannequin arm while keeping the digital burn in place on top of it. To achieve this, our software engineers took a 3D scan of the mannequin arm and used it to train a computer vision model to recognise it in real-life. Once the arm is detected, the digital burn can be seamlessly overlaid on top of it, matching its size and shape.

The software team redefined the 3D scan and created high fidelity digital burn wounds directly on top of the arm. The phone app that we used allowed us to manipulate the digital overlay by selecting various skin colours and types of burn wounds for assessment during training.

Because the training surgeon can interact with the physical mannequin arm while using the mixed reality training app, we ensured that the prototype maintained good hand occlusion. This was crucial to avoid obstruction of both the digital overlay applied on the mannequin and the hands of the training surgeon. Maintaining good hand occlusion also provided a good level of realism and ultimately, a better training experience.

Outcome

The mixed reality solution we developed for Buckinghamshire NHS Trust substantially expands the exposure of training surgeons to diverse burn scenarios, improving their learning outcomes and ultimately, patient care. This solution significantly reduces training costs and resource requirements for the NHS. The technology was de-risked and proven to be feasible and our client plans to develop it further into a commercially available product.