Dear Community (and Developers), for a research project at my University I need to collect participants’ eye tracking data (along with other measures) in a Virtual Reality environment that I have created in Godot 4.2. As VR headset I have an HTC Vive Pro Eye that has this feature. Moreover, I found that this functionality is supported also in OpenXR and some APIs are available for other game engines. However I didn’t find a toolbox or an API that could connect and support this functionality in Godot and to date I’m unable to collect this kind of data. In Godot’s official website I only found a news of September 2023 (Godot XR update - September 2023) reporting this: “Eye tracking support is nearly finished but is awaiting some enhancements regarding permission requirements”. @mux213 are there any news? Can I have the possibility of collecting the eye tracking data in Godot at present (my project will start in weeks)? Otherwise, would it be possible to develop this type of support in some way? Many thanks in advance!
I haven’t been able to test anything specifically with the HTC Vive Pro Eye as I don’t have access to the hardware, however the standard OpenXR eye gaze interaction API should be available in Godot 4.2
You do need to do some setup here.
In your project settings, under OpenXR, make sure that the eye gaze interaction extension is enabled
In the action map editor, add a pose action called “eye_gaze”, then add a profile for the eye gaze interaction to your action map. Don’t forget to save the action map.
Now add an XRController3D node to your XROrigin3D node, set it’s Tracker to /user/eyes_ext and it’s Pose to eye_gaze:
If all is setup correctly the XRController3D node should be placed between the two eyes, and be oriented so it’s Z orientation points to where your user is looking.
Hi Bastiaan, as you suggested, I tried these configurations with the HTC Vive Pro Eye and SteamVR always on.
To check if everything was working, I printed the Transform coordinates (position and rotation) of the XRController3D node (named EyeGaze) frame by frame, but I always received 0 for all the axis. Maybe the eye gaze data are flowing elsewhere? By chance, are there any other settings to adjust?
In the XRController3D node Inspector, as you can see by the screenshot below, I had to manually write “eye_gaze” in the Pose section because for the “/user/eyes_ext” Tracker only the “default” pose was present in the drop-down menu. Could this be the main problem? How could this be solved?
Thanks in advance
Indeed, you need to manually enter that, at some point we need to extend the inspector to be able to read from the action map.
One thing to double check is to turn on verbose output in your project settings and then run your app and see if the eye gaze interaction is reported as supported.
Looking at the list of runtimes that support it, only the standalone HTC headsets are listed and Valve is missing from the list: https://github.khronos.org/OpenXR-Inventory/extension_support.html#XR_EXT_eye_gaze_interaction
So it’s possible they never added support for it.
Thanks Bastiaan, here’s the line of the output regarding the eye_gaze_interaction:
“OpenXR: Interaction profile /interaction_profiles/ext/eye_gaze_interaction requires extension XR_EXT_eye_gaze_interaction”
But it’s strange, because the HTC Vive Pro Eye headset, along with SteamVR and SRanipal, seems to have the OpenXR eye tracking support according to the documentation, at least for Unity and Unreal:
That message indeed indicates the eye gaze interaction has not been enabled and thus the action map entries are being filtered out.
With verbose on it should give a list of all supported extensions at startup. This would tell you if the extension is not supported at all, or if it is not enabled.
It’s possible that Unity and Unreal aren’t going through OpenXR but some other API to get to the data.
Ah, I got some info from Matthieu, looks like you need an OpenXR layer installed that adds the eye gaze support. He ran into similar issues and did a writeup about it:
So its about making sure the eye tracking layer is enabled in SteamVR and fully calibrated. Then Godot should be able to access it.
Thanks Bastiaan, I followed Matthieu’s instructions and I have some updates.
The good news is that now the XR_EXT_eye_gaze_interaction extension was found and enabled. The bad news is that Godot now runs in a debug mode and everytime I run the project, the engine stops before opening the VR window. I didn’t change anything in the debug menu (here’s the screen):
I’ve attached the whole log of the terminal below. The android related error occurs because I’m developing the environment also on the PICO VR headset, but these logs come from the HTC Vive Pro Eye execution:
Running: C:/Users/user/Desktop/Godot_v4.2.1-stable_win64.exe
–path
C:/Users/user/Documents/XR%20Tutorial%20Godot%204.2_v0.14
–remote-debug
tcp://127.0.0.1:6007
–editor-pid
10904
–position
704,360
res://scenes/main.tscn
ERROR: Error loading GDExtension configuration file: res://android/plugins/meta/plugin.gdextension
at: load_gdextension_resource (core/extension/gdextension.cpp:860)
ERROR: Failed loading resource: res://android/plugins/meta/plugin.gdextension. Make sure resources have been imported b.
at: (core/io/resource_loader.cpp:274)
ERROR: Error loading extension: res://android/plugins/meta/plugin.gdextension
at: (core/extension/gdextension_manager.cpp:234)
Godot Engine v4.2.1.stable.official.b09f793f5 - https://godotengine.org
TextServer: Added interface “Dummy”
OpenXR: Found OpenXR layer XR_APILAYER_MBUCCHIA_quad_views_foveated
OpenXR: Found OpenXR layer XR_APILAYER_VIVE_facial_tracking
OpenXR: Found OpenXR layer XR_APILAYER_ULTRALEAP_hand_tracking
OpenXR: Found OpenXR extension XR_VARJO_quad_views
OpenXR: Found OpenXR extension XR_VARJO_foveated_rendering
OpenXR: Found OpenXR extension XR_EXT_eye_gaze_interaction
OpenXR: Found OpenXR extension XR_HTC_facial_tracking
OpenXR: Found OpenXR extension XR_EXT_hand_tracking
OpenXR: Found OpenXR extension XR_EXT_hand_joints_motion_range
OpenXR: Found OpenXR extension XR_EXT_hand_tracking_data_source
OpenXR: Found OpenXR extension XR_MSFT_hand_interaction
OpenXR: Found OpenXR extension XR_FB_hand_tracking_aim
OpenXR: Found OpenXR extension XR_ULTRALEAP_hand_tracking_forearm
OpenXR: Found OpenXR extension XR_KHR_vulkan_enable
OpenXR: Found OpenXR extension XR_KHR_vulkan_enable2
OpenXR: Found OpenXR extension XR_KHR_D3D11_enable
OpenXR: Found OpenXR extension XR_KHR_D3D12_enable
OpenXR: Found OpenXR extension XR_KHR_opengl_enable
OpenXR: Found OpenXR extension XR_KHR_win32_convert_performance_counter_time
OpenXR: Found OpenXR extension XR_EXT_win32_appcontainer_compatible
OpenXR: Found OpenXR extension XR_KHR_binding_modification
OpenXR: Found OpenXR extension XR_KHR_composition_layer_depth
OpenXR: Found OpenXR extension XR_KHR_visibility_mask
OpenXR: Found OpenXR extension XR_EXT_active_action_set_priority
OpenXR: Found OpenXR extension XR_EXT_dpad_binding
OpenXR: Found OpenXR extension XR_EXT_frame_composition_report
OpenXR: Found OpenXR extension XR_EXT_hp_mixed_reality_controller
OpenXR: Found OpenXR extension XR_EXT_local_floor
OpenXR: Found OpenXR extension XR_EXT_palm_pose
OpenXR: Found OpenXR extension XR_FB_display_refresh_rate
OpenXR: Found OpenXR extension XR_HTC_vive_cosmos_controller_interaction
OpenXR: Found OpenXR extension XR_HTC_vive_focus3_controller_interaction
OpenXR: Found OpenXR extension XR_HTC_vive_wrist_tracker_interaction
OpenXR: Found OpenXR extension XR_MND_headless
OpenXR: Found OpenXR extension XR_VALVE_analog_threshold
OpenXR: Found OpenXR extension XR_HTCX_vive_tracker_interaction
OpenXR: Found OpenXR extension XR_EXT_debug_utils
OpenXR: Enabling extension XR_EXT_palm_pose
OpenXR: Enabling extension XR_KHR_composition_layer_depth
OpenXR: Enabling extension XR_HTC_vive_cosmos_controller_interaction
OpenXR: Enabling extension XR_HTC_vive_focus3_controller_interaction
OpenXR: Enabling extension XR_HTCX_vive_tracker_interaction
OpenXR: Enabling extension XR_FB_display_refresh_rate
OpenXR: Enabling extension XR_EXT_hp_mixed_reality_controller
OpenXR: Enabling extension XR_EXT_eye_gaze_interaction
OpenXR: Enabling extension XR_EXT_hand_tracking
OpenXR: Enabling extension XR_EXT_hand_joints_motion_range
OpenXR: Enabling extension XR_KHR_vulkan_enable2
[2024-05-08 16:15:05.748] [info] [Config.cpp:59] API layer configuration file does not exist, skipping
[2024-05-08 16:15:05.748] [info] [PlatformContext.cpp:45] Loaded configuration variables.
The GDExtension related errors you can ignore, we don’t use those in this scenario.
The important ones I think are the last two but I’m not sure whats not setup right here, whether we are missing something in setting up layers, or if this is something that is missing on Vive side.
I did get some feedback that some of the layers Vive makes available can cause crashes in certain XR clients. You could try and make sure the eye interaction one is the only one enabled and see if that changes anything.
Now the eye tracker data seems to flow correctely without errors. I only have to check if calibration works before running the game, but we’re almost there…
Ah cool, yeah we don’t use the quad view solution (yet), strange it would crash on it but you don’t need it. I am looking into support for it (have a PR up already) but it’s a pretty niche feature.
I hate pinging you @mux213, but you seem to be the only one who has answers about eye tracking.
Will Eye Tracking in Godot 4.2 work with a Quest Pro, if not will it be possible any time soon? I have the Eye Tracker extensions from Matthieu installed and working with OpenXR Tools, but I don’t see any way to get Godot to work with it.
I have a project coming up in the next month that requires Eye Tracking. My options are buy an HTC Vive Eye for development in Godot, or use my Quest Pro in Unity & I really don’t want to use Unity or buy a Vive Eye… but if a Vive Eye is my best option then so be it!
Please let me know if Quest Pro is on the road map, or if the Vive Eye is my best option for Eye Tracking in Godot.
I can tell you for sure that the eye tracker of the HTC Vive Pro Eye now works in Godot with the setup that we “discovered” in this thread. Thanks to @mux213! Don’t know exactly if the same setup works also for other headsets…
Eye tracking for the Quest Pro works, I think it may work in 4.2, but definitely in 4.3. Indeed all through the eye gaze interaction profile. I can also confirm it works with Varjo (and damn it’s so much better on Varjo). I think Malcolm also confirmed it worked on HTC Elite XR with eye tracker, I sadly don’t have the needed add on.
Thanks for confirming it works with the HTC Vive Pro Eye @francesco.digruttola , that is good to know.
@francesco.digruttola can you edit your original post to add the XR sub category to it (see Categories - Godot Forum )? I don’t know if you can do this after a topic is created but if it’s possible, would be easier for people to find this thread.
I will continue to fiddle with the configs in 4.2, but I didn’t have any luck. I’m probably missing something somewhere.
It took way too long to find all the little places where I had to enable eye tracking for Quest Pro. The last one that got me was even though I enabled it everywhere, including beta option in the oculus PC app… I missed enabling it on the Meta Quest Mobile App! Under the Developer option!Once I got that, Matthieu’s testing tool started working
It would have taken me a lot longer to get Eye Tracking working on PC without Mathieu’s tools and instructions… so thank you Matthieu!
I’ll also give 4.3 a look and see if I have better luck with Quest Pro Eye Tracking!
Once I got that, Matthieu’s testing tool started working