The world’s first augmented reality surgical operation

A man watches with a futuristic look with glasses augmented reality in holography. [Shutterstock]

This article is part of our special report How the light gets in: Europe’s Photonics Landscape.

For Giovanni Badiali, it is a day like any other. Adorned in the standard surgical attire, he enters the operating room and prepares his medial implements with the patient lying flat before him.

There is one noticeable difference to today’s procedure, however: Badiali pulls a visor over his face and a series of digitised numbers and images appear in front of his eyes. He is about to embark on the world’s first augmented reality operation.

“We didn’t expect to get such a perfectly attuned experience to the surgical procedure,” Badiali, lead surgeon of the operation which took place at Bologna’s Sant’Orsola hospital in February, told EURACTIV.

“We had a real patient both under our hands and under our eyes, the entire experience was heightened. Suddenly I had instant and seamless access to a wide range of data about the patient, which proved to be a vital aid in the operation.”

The procedure itself involved the resecting and repositioning of the upper jaw bone, in order to restore the patient’s biting functionality. For an operation such as this, normally an external monitor would be used to analyse vital information about the patient’s condition.

This time around, with the use of a headset developed as part of the EU-funded VOSTARS (Video-Optical See Through AR surgical System) project, that very same data appears in the lens of a visor worn by the lead surgeon.

This allows for a greater focus on the operation, improved monitoring of the patient’s condition, a more timely intervention, as well as better hand-eye coordination on the part of the surgeon.

The data visible through the visor includes the patient’s heart rate, body temperature, blood pressure, and breathing rates, alongside pre-operative information retrieved from imaging technologies used beforehand, including CT, MRI, or 3DUS scans.

Good news for patients

For the patients themselves, Badiali explained, the development of such technologies can only have a positive impact on the future efficiency of Europe’s public health services. “We made sure that the patient was well informed about the changes to the nature of this operation,” he said.

“For the patient, there are only benefits to this new way of operating. During the first initial meeting, we explained to them the new devices that we were going to use, as well as all the research that had gone into developing the visor. The patient was excited to be part of the research and they felt safe because the surgical procedure was straightforward, and there was no added risk from the use of the new technology.”

The technology itself had been long in the making.

The VOSTARS initiative is led by Italy’s University of Pisa along with 12 European partners from France, Germany and the United Kingdom. For Project coordinator Dr Vincenzo Ferrari, a biomedical engineering researcher at the Department of Information Engineering at the University of Pisa, the research still holds a lot of potential for the future.

“The possibilities with this technology are comprehensive,” Ferrari told EURACTIV. “It’s now possible to superimpose images onto a surgeon’s visual perception, allowing the doctor to see with an ability that is akin to x-ray vision.”

“With the first results of the VOSTARS project, we have shown that we can overcome technical obstacles to enhance the experience of the surgeon and transmit the data that they would have otherwise had to redirect their field of vision to see,” Ferrari added.

The next steps, he said, are to incorporate a real-time video feed into the visors, so that it becomes possible for the surgeon to follow the movements of implements equipped with cameras as they guide surgical procedures.

“This could help to create perfect alignment between the actions of the surgeon and the data transmitted as part of the footage,” Ferrari said.

On the edge of medical revolution

The technology that provides the basis for such ground-breaking leaps in surgical operations is based on the use of photonics applications. EURACTIV spoke to Anna Pelagotti from the European Commission’s Photonics’ Unit, who has been following the VOSTARS initiative.

“Imaging devices are used throughout the consultative and pre-operative phases of health service procedures, in order to plan surgeries. What this project does is merge the data analysed during such stages with the surgical stage itself so that the doctor is able to analyse all of the information relevant to the operation,” she said.

“Photonics is the science of light technology, as we define it. It includes not only the transmission of light but also the acquisition of light,” Pelagotti said. “The display used in the VOSTARS project featured many photonics components, which are absolutely vital for these technologies to function.”

Pelagotti added that the project such as the one recently conducted in Bologna is just one of the many others currently established under the Public-Private Partnership (PPP) arrangement between the EU Commission and the European Technology Platform’s Photonics21 Association, within the Horizon 2020 funding framework.

Other ongoing projects include Prometheus, which aims to deliver the next generation in high power ultra-short pulse laser surface processing, and the Galahad initiative, which vies to develop photonics technologies to better detect the onset of age-related visual impairment diseases.

Reflecting on his performance in the operating theatre and being the first surgeon in the world to conduct an augmented reality operation, Giovanni Badiali told EURACTIV that the outcome of the procedure spelled positive things for the future of advanced technologies in hospitals.

“The boundaries of what we can introduce to the surgical field are limited by just our creativity. It’s possible to add datasets that relate to fields of anatomy, pathology, pathways, and pathfinders,” he said. “We are on the edge of a medical revolution in terms of surgical navigation.”

“We could be working seamlessly in the future. For surgeries, this means a great gain of time and a reduction of mental work to do the connections between the virtual and the real. All the information arrives in real time. Time is the greatest beneficiary of this technology.”

[Edited by Zoran Radosavljevic]

Subscribe to our newsletters