Godot 4 - AR development

Godot Version

Godot 4.0.2

So I’ve been doing some investigation since I have a subject I have to teach about AR/VR/XR this year, historically previous teachers were using Unity, and the idea for this year was to go to Unreal Engine, however while I really like UE, I would prefer to shift towards Godot, basically because it’s happening the same as it happened with Blender back in the day, I think it’s becoming a very powerful professional tool that will evolve more as more users it has, and that it’s better to teach an open and free tool like Godot than a preoprietary draconian tool that can change their conditions in the near future to something absurd ( as Unity), so I don’t want to act as a “drug-dealer” for them, it’s not ethical in my opinion , at least for me, not judging others.

So the question is that I’m trying to find information about AR development for mobile devices, but all the information I found is kind of outdated or the support is not there.

So can someone help me with this please?

Thanks!

P.S.: if this is not the correct forum to place this thread, move it right away or tell me where to post please :slight_smile:

There is pretty good support for standalone Android / OpenXR-based XR headsets in Godot.

We also have 6+ years experience using Godot for AR in multiple different environments (custom automotive and headset based), so it is possible to use Godot for AR, but possibly not out of the box.

I am unsure about the current ARKit / ARCore support in 4.2 / master, probably Bastiaan (@mux213), the XR lead maintainer knows more about it.

Also, If you post the curriculum that you want to teach and what kind of hardware you have, it will be easier to see which parts will work out of the box, and which parts would require extra development in the engine.

Kind regards,
Gergely

1 Like

The problem is that AR is a very ambiguous term these days, which is why I think Apple is trying their hardest to move away from the term and have us all talking about spatial computing when dealing with their vision for the Vision Pro.

Godot does have an outdated plugin for ARKit for Godot 3, and we’ve got a half finished ARCore implementation that we’ve never gotten to work properly (though I hope one day someone with the right experience will come along and give us a nudge to make it work, I do believe we are very close). This sort of phone based AR seems to slowly be going the way of the dodo. If this is to have a future I do hope some day vendors like Google and Apple embrace OpenXR.

AR in the sense of AR glasses however is seeing more and more focus. Many vendors have now enabled this through passthrough (though I feel this is a stepping stone technology, but that is my personal opinion not everyone agrees with :slight_smile: ), as well as real AR solutions. This is something where we have decent support in Godot with an ever growing focus and more and more contributors (like Kisg up above) making great strides in improving this.

We support AR through OpenXR with many passthrough headsets like the Quest, Pico, HTC, Lynx R1 and real AR like the Magic Leap 2,
we also support TiltFive AR glasses through a separate plugin,
and hopefully some day soon Vision Pro, but no guarantees there, there is a lot of ground to cover still.

Finally there is one alternative where I do not know the status of, but I do know people were experimenting with it. Phone/Webcam AR through WebXR is an option worth looking into, though I think Godots support here is likely fairly basic. Something that @dsnopek might be able to shed some light on.

1 Like

Well is not that ambiguous, the generic accepted terminology (at least in the circles I move) is easier than what it seems:

VR - Virtual Reality - Headset on face, FOV isolation from real world

XR - Mixed Reality - Headset on face, FOV augmentation of real world

AR - Augmented Reality - Handheld device - augmentation on device, in hand, no relation to FOV

I think if this differentiation is made clear in documentation and the lack or ARKit and ARCore support is clear it will help many people clarify this, because I’ve seen many questions like this over the internet but not a clear centralized place with this information.

Correct me if I’m wrong but what I understand is:

VR - Supported in general, through OpenXR

XR - Supported in general, in the same way as VR, through OpenXR

AR - Not supported at all, existing outdated plugins or implementations that are not finished, not available at the moment

Thanks for the info, the pity is that I won’t be able to teach my subject with Godot, I may slip it in the final part.

Basically the subject is about AR/VR/XR in general, is for art driven people so I deliver kind of a template for each case that they will use for their projects.
I wanted to avoid Unity or Unreal, but it seems not possible in this case, I’m kind of angry at the situation (not at Godot or anyone else :slight_smile: ) but I think for the time being it is what it is :slight_smile:

Thanks for the help!

Phone/Webcam AR through WebXR is an option worth looking into, though I think Godots support here is likely fairly basic. Something that @dsnopek might be able to shed some light on.

AR support in WebXR is pretty basic. Rendering works, both on stereo devices (like Meta Quest in mixed reality mode) and mono (like on smartphones). But we haven’t implemented much in the way of AR-specific features, like anchors and hit testing. (I’m hoping to add hand tracking for Godot 4.3, but we’ll see)

1 Like

Thanks for that info, I left WebXR out of the equation kind of on purpose precisely because of the artistic limitations it has, but it’s good to know it has basic support for mono devices, I’ll try to do some tests when I have a bit of time :smiley:

Thanks!

@juangea , actually I’m afraid that isn’t a very accurate description of the various terms.

Hololens, NReal, TiltFive and Magic Leap are all very clear AR devices and some of these have been around longer than phone based AR.

AR, indeed Augmented Reality, has always been a term to mean that you overlay virtual elements over the real world, you are augmenting the real world. The method by which you do this is not important, whether overlaying on real world camera footage (phone AR and passthrough AR), or by projecting an image in front of the players eyes, it’s all AR.

Phone based AR was mostly introduced as a stepping stone and there is very little focus on this by the industry with only Apple really pushing it further but mostly to get people ready for the Vision Pro, so I think ARKits days are numbered.

I do think this is a mistake, I think phone based AR has many use cases and I wish we had better support for it. But it has been hard finding people with an existing skillset who wish to spend their free time on implementing this and without industry interest it’s hard to find funds to get paid people to do this.

XR was designed as a catchall acronym. OpenXR was named as such to make it clear that it covers all devices, the X basically standing in for any letter and not really meaning anything (though some have adopted it to mean eXtended).

Again Hololens being one of the first to adopt OpenXR (Microsoft was a very early adopter of the standard) as an AR API, sadly Hololens being UWP and DirectX only we never managed to get Godot running on it.
Magic leap adopted OpenXR for their Magic leap 2 headset, again a full AR headset that runs Godot through OpenXR :slight_smile:
NReal is an unknown at this time but rumoured to have an OpenXR API in the works (or possibly released) so hopefully we’ll be able to run Godot on that as well.
TiltFive is the other exception as their feature set isn’t well covered by OpenXR (yet), but we support them through a separate plugin.

The problem OpenXR right now has is that while the core feature set for AR is there, many of the features found in ARKit and ARCore are still missing from the spec. I believe even Microsoft kept some of their features part of their proprietary APIs. There are various vendor extensions being added but AR support in OpenXR needs more maturing. I do think this will ramp up as companies like Magic Leap, Varjo, Meta and Lynx are all investing heavily in making AR happen.

Sadly the term XR also has the problem that it has been appropriated by the eXtinction Rebellion movement, while I won’t cast judgement on that movement, it has resulted that if you search for XR on social media or google, you’re unlikely to find topics that has anything to do with VR or AR.

Finally MR - Mixed Reality, which is probably the most confusing term of all because it means different things to different people.

Microsoft started using it to refer to their Windows Mixed Reality platform because that platform supported both their AR device (the Hololens) and VR devices (Acer, HP, Lenovo, etc. headsets) with the hope that people would adopt their VR devices as a stepping stone to creating AR applications for the Hololens.

At the same time Oculus and a number of other developers of similar solutions adopted MR to mean combining video of the person wearing the headset, with a matching virtual camera view to create a 3rd person view of the game being played which is a much more enjoyable experience for spectators or when capturing footage of someone playing a VR game (see tools like Liv or Mixcast).

Finally indeed, MR was adopted by some as a catch all phrase, possibly because they didn’t like XR as a term.

I can really see why Apple wants to move away from the terminology and is calling it spatial computing now.

1 Like

Owh and before I forget, phone based AR is fully supported in OpenXR, it was part of the earliest specs. It is just that no phone vendor has implemented support for it.

I would love to see an OpenXR runtime for phones that unlocks this. Now that Android support for OpenXR is maturing, hopefully someone will one day make this a reality.

1 Like

Thanks for the detailed answer, I really enjoyed reading it and I get your points.

After reading you, I agree, the idea I had is not the correct one, and it’s a complex field.

I think VR meaning is clear, but also AR then means both handheld or “whatever-held” as long as it places 3D over real world.

The problem with this is what you stated, AR in phones is not supported and that can lead to confusion to people looking for Godot capabilities, it took me a while to understand the current situation.

In communication and company events AR is actually used a lot, of course if it is WebAR is most welcome but AFAIK on iPhones that’s not supported, and that’s what a big amount of executives has, so it has to be supported, that’s why ARKit ends up being the option.

(To be clear, I refer basically for internal company events or limited distribution for fairs)

Well, nice conversation, sadly without being able to present things on the phone I’m afraid I won’t be able to teach the subject with Godot, I’m not entirely sure, I’m still looking into it, but we’ll see :slight_smile:

Thanks!!!

Fully agree there Juangea, I wish this was something we had funding for or found experienced contributors for to make sure its maintained, and who knows that may still happen.

It is indeed really weird that there seems to be a really active following for phone AR, yet there seems to be very little focus on it from the industry. Everyone just wants to go for the glasses medium and while that definitely has a big future, there are still use cases for phone AR.

1 Like