Multi-modal input for XR

[Jack_C]

Godot Version

4.4.1

Question

Hey, I’m wondering if Godot’s XR supports multi-modal inputs with one hand using optical hand-tracking, and the other using a regular controller? Similar to what this article describes:

Use Simultaneous Hands and Controllers (Multimodal)

I’m trying to implement it in a game where left hand is used with hand-tracking to recognize gesture to cast spells while the right hand uses controller’s thumbstick to move around.

[mrcdk]

The Godot XR vendors plugin supports Meta hand tracking Meta Hand Tracking — Godot OpenXR Vendors plugin documentation but I’m unsure if you can mix both at the same time. Seeing that you need to enable it with an extension it’s probably not supported but maybe try and see? it may work. If it’s not working you could try opening an issue and see what can be done.

[mux213]

I think we already discussed this on the Discord server but just to have this information recorded so people can find it.

From the point of view of how Godot has implemented hand tracking and the action map system, there is nothing preventing this from working. The setup is perfectly capable of switching between hand tracking and controller tracking and makes no assumption that the both hands act the same.

Meta however has a backwards compatibility issue, introducing the ability for the left hand having hand tracking and the right hand having controller tracking while existing applications may not cater for this, is a problem. So they made this switchable and something you have to opt into.

The XR_META_simultaneous_hands_and_controllers extension enables this however we currently do not have this implemented. This is a vendor extension that has not (yet) been accepted into the OpenXR spec and falls under Metas SDK license. Not an unsurmountable issue but it is a barrier.
Implementing the extension itself should be relatively easy.

The other extension mentioned here, XR_META_detached_controllers, has the same issue that its not in the standard spec and is equally fairly straight forward to add code wise, but whether this one is necesairy is another thing. We already support XR_FB_render_model which allows visualising the correct controller whether that controller is held or not.