Cognitive and Perceptual Effects of Cross-Modal Illusions in Augmented Reality

Abstract

This research project aims to explore to which extent Augmented Reality (AR) experiences can evoke measurable physiological, psychological, and neurological responses in the user’s body.

We have developed an AR experience that enables users to see and hear their own hands burning while looking through a Video See-Through Head-Mounted Display (VST-HMD).

We have conducted a series of three experiments so far. The first experiment examined the prevalence of heat illusions and physiological and psychological stress responses induced by your AR experience. Half of the participants reported a heat-sensation on the affected left hand. All participants experienced a significant increase in skin conductance during the experiment. Moreover, participants who experienced a heat sensation had a higher skin conductance response.

The second experiment investigated whether a heat illusion would lead to thermoregulatory responses. Using a Laser Doppler Flowmeter, we demonstrated that skin blood flow in the affected hand changed significantly in some participants. In a follow-up study, we showed that this experience also affected hand microcirculatory blood flow in some participants.

Experiment three examined whether this AR experience could manipulate participants’ perception of pain. The virtual flames on participants’ hands led to a decrease in the temperature at which they perceived heat-related pain. These results suggest that AR-induced stimuli may be an effective method for top-down modulation of the pain experience.

The main conclusions of the first three experiments can be summarized as follows: (1) AR can reliably evoke cross-modal illusions in subjects through visual and auditory stimuli. (2) Involuntary heat illusions in AR can lead to some thermoregulatory responses, such as an increase in blood flow in the skin. (3) AR may strongly affect pain perception.

Our research will continue with several planned follow-up experiments to further explore how AR experiences can have measurable effects on the user’s body and mental state. This will provide deeper insights into the perceptual and cognitive effects unique to AR experiences. Findings from our experiments could be relevant in a neuroscience or medical context.

Publications

[1] D. Eckhoff, A. Cassinelli, and C. Sandor, “Heat Pain Threshold Modulation By Experiencing Burning Hands in Augmented Reality,” in To Appear In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct. 2021, p. 2. doi: DOI 10.1109/ISMAR-Adjunct54149.2021.00079. PDF

[2] D. Eckhoff, “[DC] Psychophysical Effects of Augmented Reality Experiences,” in Proceedings – 2021 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops, United States, Mar. 2021, pp. 739–740. doi: 10.1109/VRW52623.2021.00252. PDF

[3] D. Eckhoff, C. Li-Tsang, G. Cheing, A. Cassinelli, and C. Sandor, “Investigation of Microcirculatory Effects of Experiencing Burning Hands in Augmented Reality,” in Proceedings – 2021 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops, United States, Mar. 2021, pp. 569–570. doi: 10.1109/VRW52623.2021.00167. PDF

[4] D. Eckhoff, A. Cassinelli, T. Liu, and C. Sandor, “Psychophysical Effects of Experiencing Burning Hands in Augmented Reality,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Cham, 2020, vol. 12499 LNCS, pp. 83–95. doi: 10.1007/978-3-030-62655-6_5. PDF

[5] D. Eckhoff, A. Cassinelli, and C. Sandor, “Exploring Perceptual and Cognitive Effects of Extreme Augmented Reality Experiences.” PDF

Contact

Daniel Eckhoff