Video-based Side-Channel Attack in Wearable(VR) Devices

Project Overview

A video-based side-channel attack, Hidden Reality (HR), shows although the virtual screen in VR devices is not in direct sight of adversaries, the indirect observations such as hand gestures might get exploited to steal the user’s private information. The Hidden Reality model can successfully decipher an average of over 75% of the text inputs.

Hidden Reality Model

The steps involved in the implementation of the Hidden Reality attack model for various attack scenarios follow.

• Video Preprocessing
• Localization and Hand Landmark Tracking
• Click Detection
• Character Inference
• Word Prediction

Datasets

With the approval of the university's IRB, videos of registered volunteer participants were captured while they were inputting the text on their virtual screen of the Meta Quest 2. During this experiment, a large corpus of 368 short video clips was recorded during the following attack scenarios - Password entry, Pin entry, Graphical-lock pattern entry, Text entry, and Email entry.

Related Works

I Know What You Enter on Gear VR

The attack model by Ling et al. predicts the passwords typed by users by utilizing 3D video recordings of the headset and videos of fingertip taps on the touchpad of the Samsung VR headset.

Face-mic: inferring live speech and speakeridentity via subtle facial dynamics captured by ar/vr motion sensors

A motion sensor-based speech eavesdropping attack referred to as Face-mic that infers highly sensitive information from live human speech, s speaker gender, identity, and speech content.

A Keylogging Inference Attack on Air-Tapping Keyboards in Virtual Environments

A key inference attack on in-air typing on AR devices which utilizes the inbuilt motion sensors data from the AR device to track the user’s hand movements.