VR world in Godot

[64jcl]

Ok so I spent quite some time developing a VR world in Unity a few years back but felt the compile and test cycle was very slow and at one point I stopped that project. I then discovered Babylonjs which also supported WebXR and in a couple of months I had ported and added many more features for my VR world builder thingy. But the moment I started working more with rich outdoor scenes I got into trouble with performance and I had to seriously reduce the visual fidelity in general but still struggled with the engine not handling many objects or draw calls. No doubt because it was single threaded JS and it seems the WebXR translation layer just added a lot of overhead even with thin instances (which is how you speed up drawing considerably for outdoor flora).

I had read a lot about Godot and saw some dev videos, mostly pixel graphics side scrollers. I never really thought that Godot had come this far in 3D rendering until I started experimenting and saw that it supported VR brilliantly and had every bells and whistle I could want. Naturally I dove into the outdoor rendering immediately and have been gradually migrating part for part from my Babylonjs version to Godot except my server connection and house construction parts. I want to improve both anyway so those come last anyway in my Godot adventure.

I have enclosed some screenshots below to showcase some of the things. Everything is running at 72hz stable on my 4070 Ti Super with MSAA 4 on my Quest 3.

This image above was the first I took when I had working splat maps in my terrain which added a needed level of texture to the world. Everything is infinitely procedurally generated at this point with 32x32m chunks added and removed as you move about in endless exploration. The plan however is to have limited worlds/planets generated on the server to which all the things happen on a particular setting/scenario/quest/whatever (still so many options to choose how to build this).

Here I am showing the glass UI I made where the UI elements are physical objects as I simply hate the flat screens everywhere in many UIs made for VR and just wanted to experiment some. Atm the bits that are ported is where you can select objects from templates and spawn them into the world (some in the background there). I have made it somewhat generic so that I can grow the features to the UI as needed.

All objects have behaviours that can be attached to them so that e.g. the Arcade here can connect to a server that streams the arcade to the screen and sends input back to the same server as the user controls the joystick or presses buttons on the actual 3d model. At least that worked in my Babylonjs version but my Godot version above I have not wired it all up yet.

Playing a bit with two cloud layers I now have a working weather system that makes it transition between weather configurations that affect everything from lighting to wind speed (affecting cloud movement, all flora and water). The clouds are simple noise textures though as volumetrics is the most taxing part to render which I use for the fog when the weather goes stormy as well as misty mornings and evenings. The atmosphere this adds is amazing in VR.

The latest addition to the world is smaller bodies of water that can fit within a 32x32m chunk. It adds some nice elements and I am soon going to add some flora that likes in and around water like reeds to make them look a bit better.

So many things I want to do now going forward and I just want to say that Godot has been amazing to work with and is giving the performance I was seeking for my VR project as well. Hoping to post some updates here as I go along if that is of interest to people here. Please dont ask for binaries or anything, as it is in no shape to be shared at this point and besides the goal is for it to connect to a server for multi user access at one point too. But feel free to ask how things are done.

PS: Getting weather and all the dynamics to work feels like an analog synth with 100 knobs to turn so you can spend days just tweaking this down to how much motion grass should have vs a tree trunk. There is still a lot I want to do and try out there but I now try to balance the time I spend on the outdoor areas vs the actual VR mechanics and stuff like having a real avatar which is sorely missing and is a big can of worms as well.

[soapspangledgames]

The static visuals look fantastic. Good work!

[arassec]

This really looks great!

Keep up the good work and have fun with your project!

[64jcl]

Two new “behaviours” as I call them (scripts you can attach to objects) has been ported from my Babylonjs implementation. First the light behaviour, an important one. Fortunately Godot is so much better at handling arbitrary lights added to the scene that this was a breeze to add. The light behaviour naturally have an on/off state and action so can be triggered directly in hand or remotely if needed.

Secondly is the photo behaviour that lets me add cameras where the user can take pictures with them and they come out of the camera as a polaroid. The green box in front (which is now transparent glass instead) is the slot system and lights up whenever an object it can hold is near. The camera spits out the image in front in this case and it is then grabbed by the slot so that it hangs in front there until the user grabs it from the slot. It is a general system I have added that is easy to use for any object that can be slottet into others. I first made that for my Pacman arcade as I wanted the user to actually be able to slot coins in the arcade.

Btw, the vast majority of meshes are from SketchFab so all credits go to the people who made those. Except the Pacman arcade which I have modelled myself in Blender. I generally have to massage the models from SketchFab quite a lot though and in the case of the grass I have been tweaking the grass a few times, still not satisified with it but I had to stop at one point lol. While I often use convex colliders auto generated in Godot for many objects I also have the ability to make custom colliders in Blender from simple primitives too which might be faster to evaluate by the physics system. That and nodes for e.g. grip is parsed from the GLB files when read into my system, same with e.g. the flashlight where the light ends up in the node called “light” if it exists, otherwise it can be set directly in the objects template info. Finding a good workflow for adding models has been a focus but they still need to go through many hoops before I have them alive and kicking in my VR world.

A whole day off tomorrow - so no doubt there will be some more work on this project, but the todo list is getting longer by the day lol. So many things I want to try out.

[64jcl]

Added an enhancement to my light behaviour so that it can vary intensity over time to emulate the flickering of fire like from this candle. I also added a new particle behaviour but instead of specifying tons of parameters in the json I have opted to instead create one scene per particle effect and just set which type to use in my objects json config as well as the emitter point. I will likely add some extra config though for the most important things I want to tune per object which is easy to add later as needed.

That image also shows the ClockBehaviour where any clock that has two handles can reflect either real time or game time (a mode you can switch through the actions on the object).

As for the candle particle effect I use a low lifetime and a bit high gravity but with local coords turned off so that no matter the tilt of the candle the flame goes up. However I’d really like the flame to stick better to the candle while moving. I tried setting inherit velocity to 1 but no luck there. The particles have no emitting velocity as they should basically be only affected by gravity. If anyone has any tips that would be great.

I have also added a wind sensitivity behaviour so that fast motion of the candle (waving or running) will put out the light/particles. In addition, the wind speed and direction affects the particles as well just for some added realism. Tweaked my foliage shader to also have wind based lean based on strength and direction in addition to the sway that was already based on wind direction.

I think the next step is to play with some rain particle effects. Or perhaps work some more on the environmental sounds to better match the wind speed. Atm I only have day time and night time env sounds playing so more sounds would enhance the feeling a lot. That reminds me that my water ponds desperately also need some kind of water sounds.

[64jcl]

Building a rich environment to explore requires quite a few bits in place. Here is a short summary of my procedural world:

• Four passes of noise, bigger, smaller, mountainous/peaks, flattening (smaller plains)
• Pseudo-Random Seed based so one number always generates exactly the same world down to individual grasses
• Chunking system - I use 32x32m chunks - each chunk has terrain and each type of flora with distance for when they become visible/fade in
• Splat Maps using 4 textures - splat map is generated from simple rules
• Foliage system that uses both grid and random spread per mesh
• Naturally using MultiMeshInstance3D for all to have few draw calls
• Each mesh in foliage has different shaders I use global variables to control animation
• Two cloud layers using simple noise textures
• Weather system that controls fog, clouds, and animation of foliage + rain particle system
• Smaller bodies of water with 2 x noise textures + low res light probe for reflection
• No shadows on smaller foliage, only bigger ones
• MSAA 4 for little antialiasing - still some shimmering on tree foliage
• Shadow with 4 splits
• SSAO for extra shadows

Some experiences:

Especially volumetric fog is amazing but very costly in VR, I will likely combine it with the simpler depth fog and reduce distance on volumetric to be for closer areas.

Foliage fade in for e.g trees is brutal since the auto-lod system removes outer bits where the leaves are meaning you have skeleton trees in the backround. Will swap this with cross-quads and have my own LOD for those since I swap in all for a full chunk anyway.

Many meshes is surprisingly performant - I have up to 1500 instances of grass per chunk each with 25 quads or so using alpha clip (never use alpha blend). An advice is to focus on reducing overdraw cost - make your alpha clip meshes be filled by as much as possible of the image, every clear pixel = extra cost. Going from 1000 to 1500 instances for my grass really brought the scene alive and with a simple vertex shader that adds motion it feels very nice.

Atm I have way too little variation in my flora meshes, so I will gradually add more as I go. If you start studying even a game like Witcher 3 you will see how simple many of them are, plain billboards often, but the volume of them makes it come alive. Same with Skyrim. If you study e.g. the meshes people make for modded Skyrim you will see what kind of budget they operate for on that older engine that actually still looks rather amazing if modded with the latest community shaders and upgrades to textures (like parallax displacement mapping).

Some things I want to add in the future:

• More environment sounds! Just the ones I have now with wind, rain, watersounds by the pond, footsteps, they add so much depth. I want to have birds chirping in daylight, crickets in the night (although not all biomes has that ofc, but this is just for fun anyway). Since I know where each tree is around you I can even trigger sounds from them - like a bird or crow. A bush can have some kind of motion as if something moved (those made me jumpy in Skyrim at times). Howling wind when you are higher up in the mountain areas!
• Thunder! Yes my rains are already feeling somewhat decent but thunder and lighting will be icing on the cake.
• Motion in smaller foliage as I walk over them - although at this point there is no feet lol
• Ripples in the water - I have sounds when you enter water but no interaction and not even an underwater shader/post-processing effect
• An actual body and not disembodied hands! This is very complex I think. I will start with a simple upper body IK rig for arms so that the elbows move somewhat correct. And then expand with a full set of feet - no doubt a bigger challenge as I’d need it to blend between animations mixed in with IK placement of feet or something like that - never done anything like that before so a big challenge.
• Spawning of cave systems - punching holes in the terrain procedurally with some simpler caves at first and more elaborate things later
• Some kind of NPC - monster or character you can interact with in some way - been playing with rigging a char in Mixamo and have been able to download some animations to that rig so I will soon play around with at least have some kind of thing moving about in some automated way
• Porting and rebuilding my house building from Babylonjs. That one had auto generated rooms for a home that you could decorate and change materials everywhere, but I want the user to actually draw down the foundation and build from that before I do greedy meshing and form rooms to plant light probes etc. So there will be some kind of edit/play mode here as you go in and out of those modes.
• Connection to my server that I used for my Babylonjs project for login and interaction with the LLM agent you had available at all times there. It was mainly just a chatbot but also with some tools access in the form of commands where you could ask it to bring up the menu or spawn objects in front of you. It was quite fun and I have hardly scratched all the things I want to do with the LLM - including integration to whatever NPC I eventually have in the world.

Yes it is a massive undertaking - but it’s also a very fun hobby project. Babysteps, one feature at a time. Godot compiles and runs the project so fast that the test cycles I have are so fast that I sometimes just leave the headset on and do stuff through passthrough on the Quest 3 at times, lol.

[64jcl]

Ok I have been adding a few things lately. First a debug screen that I can turn on and see some metrics about the performance as well as a the debug log and filtering of it.

I added trigger volume support in meshes so they fire events that behaviours can respond to - so I added a TeleportBehaviour that will teleport the player instantly to any other position.

I have tweaked the shaders so they correctly fade in/out now in the distance. I noticed I could easily bump up range as well to 200m of terrain and a bit longer with the flora meshes too. I tweaked the spawning of ridges a bit to have taller mountains which is nice. Trees are more abundant too (although no biomes so no real forests yet).

Also a few convenience options added, togging of head-steering mode. I normally dont like that but prefer to move in the direction my body is facing, not my head, but some like that to get less seasick. Seated/roomscale toggle - the base collider will shift in both modes though but with seated you have to move quite a bit away before that happens, while it is less with roomscale. Not found the good balance here as you would want to lean out to the sides or bend forward to e.g pick up stuff and it is important your body stays put so you dont move with the head.

Rain has been tweaked a bit to be a bit better - not quite there yet but having more of it and making it darker and react to light was important. When it rains in the night and you hold a light the way the rain becomes visible is very nice.

A few pictures to finish off the post.

Sometimes I just wander around to find good spots to take pictures.

Bad weather in the evenings and mornings often bring out the best volumetric fog effects. The way the valley below had light in one part and shade in another was great to experience.

A view of the rain. With the sounds of wind and rain and the motion it feels nice. They are all simple stretched quads so no fancy stuff. No collision with landscape/meshes yet though. Supposedly there is a special support for that in Godot but it needs a constantly updated elevation map thingy so it knows where GPU should collide with stuff. Hope to add that one day.

A night shot while holding a flashlight. The range is too short so I will adjust that for the flashlight, but here you might glimpse the starry sky as well. It doesnt look great in VR but it works so far. I need to dynamically adjust down the energy multiplier of the sky in the night so that my star texture can have more dynamic range (they are all very dark grey in fact here). I also need to do the same with the reflection probe intensity which now makes reflective stuff look like they have a glow from below in the night. Everything is basically lit by a blueish ambient light at night anyway. There is no moon(s) yet so no real light source in the night so that might be a better solution in the long run as it could have a direction as well then like the sun.

[64jcl]

Time for a little update to my project. A lot has been added and refined.

• Server login and agent LLM interaction. You can now chat with an agent through voice (using whisper on the server) and get replies from the agent LLM (using Gemma 4 E4B). The text is converted to speech (using Piper TTS) and you can hear the reply very fast (1-2 seconds depending on how much is in the reply). The agent has a prompt that also includes commands so that it can open up the menu, spawn objects from a list of templates, change the weather or time of day. Oh and you can use voice in the login screen although saying your password out aloud is perhaps not the best so I will try out some interesting options there soon.

• Debug panel with information so that I can see and filter the logging instances. At the moment you can only see the client logs, will add a server log for those with admin access at one point.

• Client/server synchronization of objects and players, so now its a multi user experience that can share the world and interact with objects that the others then see move about and stuff. I guess at this point two people can toss a ball to each other, and that alone is a major achievement. There are so many interesting challenges in this domain, like ownership of the physics and sending this info to the server to be broadcasted to all users. At 10 hz it is actually quite good, and with velocity we can also do dead reckoning if a players events are lagging out.

• First very crude avatar system, head, body and hands are now synchronized. This took many rounds to get right as you dont want the headset to move the player base around unless you are playing room scale in which case there is a threshold for how much motion results in a reposition of the player base.

• Server time and weather (wind speed, wind direction, weather type) so that all clients have the same environment.

• Standalone PC build so it is possible to enter the world without VR headset. At the moment there is limited interaction in this mode as I need to have new systems for targeting for interaction and such. This was a fantastic way to test multiplayer ofc and was a primary driver for me to add this.

• Non-server sandbox mode. If there is no server available the whole thing starts up client only and everything is a client sandbox. I will try to maintain this as it can be fun to just make a single-player version of the engine with whatever systems I choose to add there. I guess my biggest problem in this regard is that my server is Nodejs based, meaning that any system that should both exist on the server and the client has to be ported/maintained. I am seriously considering a .net based server instead to simplify this ofc (I am using C# as dev language for the client btw, not GDscript).

• Improvements to terrain generation so that mountains feel more varied and rugged with erosion and terracing. It still doesnt make them look good as the 1m resolution of the terrain really limits how larger elevation changes can look. I might play with lower terrain res as well as kit bashing to make vertical mountain polygons look better.

  Various improvements to weather. Better rain, cloud layer wind direction lag so that a player can see weather change from a change in wind direction in the upper cloud layer before the other tracks in and there is a weather change. Wetness effect to terrain and foliage shaders so that they feel more soaked and gentle dry up depending on weather. Tweaked fog so I use both volumetric and the simple fog type in combination and can even reduce drawing distance accordingly to save performance in bad or misty weather.

• New tree type added, an elm, a nice little addition. I try to work on adding a few new foliage things now and then to enrich the world gradually.

• New star dome! I had to add a real star dome that rotates an 8k texture that I can paint whatever stars I want. Took many hours to tweak this bit to make it feel decent so that you can spot the odd bright star still in the blue sky mid day even.

• Improvements to the UI.

Beware the creepy avatar with the face scan. I plan on playing with using face scans for the avatars but they will be static faces for quite some time though. Here you are btw looking at a “mimic” feature I added so that I could also more easily test how your own avatar would look like as it act as if it was your own avatar in front of you. So shaking hands with myself is something ofc.

A not so good image showing wetness in the world which basically just makes things darker, less rough and adds more specular, so nothing fancy but works well to convey some feeling.

The debug screen that can be turned on and is sitting on your right side. A very useful tool to figure out why things dont work as they should lol. The FPS drops big time when I bring up the screenshot thingy in the Quest, as I normally have a solid 72 hz. I plan ofc to be able to position these UI windows at will by being able to click and drag them around. Depending on what direction I take this VR experience, I can imagine a player wanting to have some panels open at all times to look at now and then.

My main UI where I can control stuff in the client. I am gradually trying to device an UI system that uses 3D objects for the elements as I cant stand the flat interfaces that I see in many VR implementations. I want it to feel more tactile. But the icons here are ofc all gen-ai stuff and not really what I want but is a functional starting point as my dev cycles are rather fast I dont really linger much on one particular problem.

A quick mocked up login screen that works for now. I made the login as a separate scene as I can then put whatever I feel like into it in time and spawn it anywhere, even in the actual world if I feel like (perhaps some starting position).

Even though the clouds are simple fast noise textures they still work pretty well I think, especially with the lower light. Here you can see some of the stars coming through the darker bits of sky as well until the sun passes down completely and you are left with a nice starry sky. Screenshot also shows that I am very close to the 13.8ms frame time budget for 72hz and indeed at times it pops above that so I need to tweak performance soon. I have a lot to do on foliage in general as so much of it is not optimized well.

The last picture is showing some of the mountains that can now be generated which are way more varied in shape. With the terracing it is actually a bit fun to see how far up a mountain you are able to get. It makes exploring the world quite a bit more rewarding. It immediately made me want to add some kind of climbing mechanic just to test that out.

[64jcl]

Been a while since my last post but I have been busy improving a lot of aspects of the engine with the most focus on a fully rigged avatar with IK controlled arms and triggering of hand animations on the main avatar skeleton. I also have improved seated mode VR so that moving your head and leaning is accurately reflected in the upper body of the avatar. I also added re-centering function, although at the moment this is manual from the controller or when your position or rotation has strayed too much. Finding a good solution to the seated vs roomscale problem has been a challenge as even when seated you might push your chair back and I feel I need som kind of detection that you have actually moved yourself to a new “center/base” so your avatar doesn’t reflect that you are leaning heavily or head turned all the time lol.

The screenshot shows the Mixamo Y Bot which I have used as my test “skin” and skeleton although I remap all bones to a humanoid profile meaning I can swap out the full avatar or indeed build up my own of different parts which is the next big advancement in the engine.

I added the ability to have multiple worlds on the server as well so that I could create this test scene which is then connected to a different “world” on the server, persisting any changes to the scene there. I added a mirror in this scene as well as a good way to check if your avatar body is correctly reflecting your actions. Ofc it feels way more immersive now that you can see your own arms and body below you compared to the old disembodied hands I used.

A number of improvement in the object creation and placement has also been done to set me up for another big update that is planned, the creation of foundations, floors, walls, ceilings, windows, doors, etc. This was actually the first thing I did in my old Babylonjs version but there it was all auto generated rooms where you could manipulate them after - all using greedy meshing to keep poly count low and have continuous surfaces which is better for lighting and shadows as well.

[zoopira]

looks like perfect