Live Action Learning

Immersing students in clinical environments is always a challenge. Pandemic restrictions, large class sizes and other obstacles make it difficult to provide sufficient real-life exposure. But with the aid of technology – and a well-constructed debriefing process – HKU nursing students are participating in virtual clinical learning that is meaningful and enriching.

Dr John Fung Tai-chun, Senior Lecturer from the School of Nursing, led the project to develop a 360 virtual reality (VR) technology, that can be used with or without a headset or goggles and is operated simply with the click of a mouse. He collaborated with the Faculty of Dentistry on the technology and worked on its application with the Centre for Applied English Studies and the School of Chinese.

The aim was to create an environment where nursing students were exposed to a multi-patient simulation scenario in which a number of activities were happening at once, similar to a hospital setting.

Students were first trained to use 360 VR through their mandatory English and Chinese language classes, where they observed group conversations by actors and zoomed in to study certain interactions in detail.

In nursing classes, students applied this learning to simulated hospital scenes based on assessed topics in clinical skills and healthcare education. Through 360 VR, they observed busy scenes, discerned irregularities and other events, and time-stamped their responses. Crucially, they underwent a de-briefing process to describe the event, unpack their feelings, and consider how to apply their learning in practice.

Competence and confidence rise

Dr Fung conducted a randomised control trial of 360 VR plus the debriefing and found that students’ clinical competence, satisfaction and self-confidence all improved. Student research assistants, who were recruited to be actors in the hospital videos, also helped in the debriefing and acted as near-peer or peer teachers.

“We have broken the physical barrier of needing goggles by putting VR in a learning management system by making it so that anyone can also use it online,” he said. “This technology goes beyond being just a digital tool. It signifies a philosophical shift towards immersive, boundary-less learning environments that can not only address educational gaps but enhance the overall learning experience.”

The technology is applicable in other disciplines and scenarios. Dr Fung has seven projects underway using 360 VR to train students in such things as endoscopic procedures and health assessment and to enable international exchanges. He also sees future potential for its use in the humanities, such as recreating historical events or exploring cultural landmarks.

Dr Fung is also pushing the technology further by using AI-generated virtual actors. He has attained a patent for ‘Apparatus and method for diagnosis prioritizing and debriefing’ in generative AI patient simulation, which provides greater flexibility for creating learning scenarios. The results are promising, with improved clinical and cultural competencies observed in students using generative AI patients.

Dr Fung’s dedication to innovation in teaching has earned him several awards, including the University’s Outstanding Teaching Award in 2023, and Teaching Innovation Award (Team) in both 2020 and 2023 as the team leader. He is also a finalist of the 2024 UGC Teaching Award.

Seeing through others’ eyes

Professor Joseph YK Chan, a medical engineer and Associate Professor of the Department of Ophthalmology, wants people to understand what the world looks like when one’s eyes are afflicted by different conditions, such as age-related macular degeneration (AMD), retinal detachment and cataracts. He has therefore developed augmented reality technology that can be used interactively through one’s phone to create a realistic, accessible and impactful experience for users.

The technology involves an app that is paired with special eye goggles that have a slot for mobile phones. The phone acts as a controller to activate different eye conditions and give a first-person perspective. For instance, users wearing the goggles might experience a dark spot in the middle of their vision as seen with AMD, or the blurry vision of astigmatism.

“I think it’s important for people to understand what these different eye diseases look like and how they progress,” said Professor Chan, who was motivated after seeing his grandfather suffer from AMD, leaving him homebound because he did not have sufficient vision to go out by himself.

He has used the technology to raise awareness among more than 4,100 elderly, caregivers of visually impaired people, and primary, secondary and tertiary education students, asking them to perform tasks that further reinforce the experience. For instance, they copy line patterns while wearing the goggles, with results that may be blurry, wavy or otherwise distorted depending on the eye condition involved.

The technology has particular pertinence for Secondary Five educators because it dovetails with biology lessons about the five senses and the drive to promote empathy. The headsets are affordable – about HK$70–80 each – and gifted to participating schools. Professor Chan has received funding support from the government’s Innovation and Technology Fund for Better Living and HKU’s Teaching Development Grant.